thordata-sdk 1.5.0__tar.gz → 1.6.0__tar.gz

{thordata_sdk-1.5.0 → thordata_sdk-1.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: thordata-sdk
-Version: 1.5.0
+Version: 1.6.0
 Summary: The Official Python SDK for Thordata - AI Data Infrastructure & Proxy Network.
 Author-email: Thordata Developer Team <support@thordata.com>
 License: MIT
@@ -63,9 +63,9 @@ Dynamic: license-file
 
 ## 📖 Introduction
 
-The **Thordata Python SDK v1.5.0** is a production-ready wrapper for Thordata's AI data infrastructure. It is architected for high reliability, strict type safety, and maximum performance.
+The **Thordata Python SDK v1.6.0** is a production-ready wrapper for Thordata's AI data infrastructure. It is architected for high reliability, strict type safety, and maximum performance.
 
-**Why v1.5.0?**
+**Why v1.6.0?**
 *   **🛡️ Bulletproof Networking**: Custom core handles `HTTP`, `HTTPS`, and `SOCKS5h` (Remote DNS) tunneling, solving common SSL/TLS handshake issues in complex network environments.
 *   **⚡ Async First**: First-class `asyncio` support with `aiohttp` for high-concurrency scraping (1000+ RPS).
 *   **🧩 100% API Coverage**: Every endpoint documented by Thordata (including Hourly Usage, Server Monitor, and Task Management) is implemented.
@@ -83,7 +83,7 @@ pip install thordata-sdk
 
 ## 🔐 Configuration
 
-Set environment variables to avoid hardcoding credentials.
+Set environment variables to avoid hardcoding credentials. **Full reference:** copy [.env.example](.env.example) to `.env` and fill in values.
 
 ```bash
 # [Scraping APIs]
@@ -93,13 +93,19 @@ export THORDATA_SCRAPER_TOKEN="your_scraper_token"
 export THORDATA_PUBLIC_TOKEN="your_public_token"
 export THORDATA_PUBLIC_KEY="your_public_key"
 
-# [Proxy Network]
+# [Proxy: Residential / Unlimited / Datacenter / Mobile / ISP]
 export THORDATA_RESIDENTIAL_USERNAME="your_username"
 export THORDATA_RESIDENTIAL_PASSWORD="your_password"
-# Optional: Set upstream proxy for local dev (e.g., Clash)
-# export THORDATA_UPSTREAM_PROXY="http://127.0.0.1:7890"
+# Optional: Unlimited (high-bandwidth) if your plan has separate credentials
+# export THORDATA_UNLIMITED_USERNAME="..."
+# export THORDATA_UNLIMITED_PASSWORD="..."
+
+# Optional: Upstream proxy when behind firewall (e.g. Clash Verge port 7897)
+# export THORDATA_UPSTREAM_PROXY="http://127.0.0.1:7897"
 ```
 
+Default proxy port is **9999** (residential); other products use different ports (see `.env.example`).
+
 ---
 
 ## 🚀 Quick Start
@@ -199,6 +205,48 @@ if status == "finished":
     print(f"Download: {data_url}")
 ```
 
+### Web Scraper Tools (120+ Pre-built Tools)
+
+Use pre-built tools for popular platforms. See [Tool Coverage Matrix](docs/TOOL_COVERAGE_MATRIX.md) for full list.
+
+```python
+from thordata import ThordataClient
+from thordata.tools import Amazon, GoogleMaps, YouTube, TikTok, eBay, Walmart
+
+client = ThordataClient()
+
+# Amazon Product by ASIN
+task_id = client.run_tool(Amazon.ProductByAsin(asin="B0BZYCJK89"))
+
+# Google Maps by Place ID
+task_id = client.run_tool(GoogleMaps.DetailsByPlaceId(place_id="ChIJPTacEpBQwokRKwIlDXelxkA"))
+
+# YouTube Video Download
+from thordata import CommonSettings
+settings = CommonSettings(resolution="<=360p", video_codec="vp9")
+task_id = client.run_tool(YouTube.VideoDownload(
+    url="https://www.youtube.com/watch?v=jNQXAC9IVRw",
+    common_settings=settings
+))
+
+# Wait and get results
+status = client.wait_for_task(task_id, max_wait=300)
+if status == "ready":
+    download_url = client.get_task_result(task_id)
+    print(f"Results: {download_url}")
+```
+
+**Available Platforms:**
+- **E-Commerce**: Amazon, eBay, Walmart
+- **Social Media**: TikTok, Instagram, Facebook, Twitter/X, Reddit, LinkedIn
+- **Search**: Google Maps, Google Shopping, Google Play
+- **Video**: YouTube (download, info, subtitles)
+- **Code**: GitHub
+- **Professional**: Indeed, Glassdoor, Crunchbase
+- **Travel/Real Estate**: Booking, Airbnb, Zillow
+
+See `examples/tools/` for more examples.
+
 ---
 
 ## 🛠️ Management APIs
@@ -226,6 +274,14 @@ monitor = client.unlimited.get_server_monitor(
 
 ---
 
+## 🧪 Development & Testing
+
+- **Full env reference**: Copy [.env.example](.env.example) to `.env` and fill in credentials.
+- **Unit tests** (no network): `pytest` or `python -m coverage run -m pytest -p no:cov tests && python -m coverage report -m`
+- **Integration tests** (live API/proxy): Set `THORDATA_INTEGRATION=true` in `.env`; optional `THORDATA_UPSTREAM_PROXY` (e.g. Clash) if behind a firewall. See [CONTRIBUTING.md](CONTRIBUTING.md#-testing-guidelines).
+
+---
+
 ## 📄 License
 
 MIT License. See [LICENSE](LICENSE) for details.

{thordata_sdk-1.5.0 → thordata_sdk-1.6.0}/README.md

@@ -19,9 +19,9 @@
 
 ## 📖 Introduction
 
-The **Thordata Python SDK v1.5.0** is a production-ready wrapper for Thordata's AI data infrastructure. It is architected for high reliability, strict type safety, and maximum performance.
+The **Thordata Python SDK v1.6.0** is a production-ready wrapper for Thordata's AI data infrastructure. It is architected for high reliability, strict type safety, and maximum performance.
 
-**Why v1.5.0?**
+**Why v1.6.0?**
 *   **🛡️ Bulletproof Networking**: Custom core handles `HTTP`, `HTTPS`, and `SOCKS5h` (Remote DNS) tunneling, solving common SSL/TLS handshake issues in complex network environments.
 *   **⚡ Async First**: First-class `asyncio` support with `aiohttp` for high-concurrency scraping (1000+ RPS).
 *   **🧩 100% API Coverage**: Every endpoint documented by Thordata (including Hourly Usage, Server Monitor, and Task Management) is implemented.
@@ -39,7 +39,7 @@ pip install thordata-sdk
 
 ## 🔐 Configuration
 
-Set environment variables to avoid hardcoding credentials.
+Set environment variables to avoid hardcoding credentials. **Full reference:** copy [.env.example](.env.example) to `.env` and fill in values.
 
 ```bash
 # [Scraping APIs]
@@ -49,13 +49,19 @@ export THORDATA_SCRAPER_TOKEN="your_scraper_token"
 export THORDATA_PUBLIC_TOKEN="your_public_token"
 export THORDATA_PUBLIC_KEY="your_public_key"
 
-# [Proxy Network]
+# [Proxy: Residential / Unlimited / Datacenter / Mobile / ISP]
 export THORDATA_RESIDENTIAL_USERNAME="your_username"
 export THORDATA_RESIDENTIAL_PASSWORD="your_password"
-# Optional: Set upstream proxy for local dev (e.g., Clash)
-# export THORDATA_UPSTREAM_PROXY="http://127.0.0.1:7890"
+# Optional: Unlimited (high-bandwidth) if your plan has separate credentials
+# export THORDATA_UNLIMITED_USERNAME="..."
+# export THORDATA_UNLIMITED_PASSWORD="..."
+
+# Optional: Upstream proxy when behind firewall (e.g. Clash Verge port 7897)
+# export THORDATA_UPSTREAM_PROXY="http://127.0.0.1:7897"
 ```
 
+Default proxy port is **9999** (residential); other products use different ports (see `.env.example`).
+
 ---
 
 ## 🚀 Quick Start
@@ -155,6 +161,48 @@ if status == "finished":
     print(f"Download: {data_url}")
 ```
 
+### Web Scraper Tools (120+ Pre-built Tools)
+
+Use pre-built tools for popular platforms. See [Tool Coverage Matrix](docs/TOOL_COVERAGE_MATRIX.md) for full list.
+
+```python
+from thordata import ThordataClient
+from thordata.tools import Amazon, GoogleMaps, YouTube, TikTok, eBay, Walmart
+
+client = ThordataClient()
+
+# Amazon Product by ASIN
+task_id = client.run_tool(Amazon.ProductByAsin(asin="B0BZYCJK89"))
+
+# Google Maps by Place ID
+task_id = client.run_tool(GoogleMaps.DetailsByPlaceId(place_id="ChIJPTacEpBQwokRKwIlDXelxkA"))
+
+# YouTube Video Download
+from thordata import CommonSettings
+settings = CommonSettings(resolution="<=360p", video_codec="vp9")
+task_id = client.run_tool(YouTube.VideoDownload(
+    url="https://www.youtube.com/watch?v=jNQXAC9IVRw",
+    common_settings=settings
+))
+
+# Wait and get results
+status = client.wait_for_task(task_id, max_wait=300)
+if status == "ready":
+    download_url = client.get_task_result(task_id)
+    print(f"Results: {download_url}")
+```
+
+**Available Platforms:**
+- **E-Commerce**: Amazon, eBay, Walmart
+- **Social Media**: TikTok, Instagram, Facebook, Twitter/X, Reddit, LinkedIn
+- **Search**: Google Maps, Google Shopping, Google Play
+- **Video**: YouTube (download, info, subtitles)
+- **Code**: GitHub
+- **Professional**: Indeed, Glassdoor, Crunchbase
+- **Travel/Real Estate**: Booking, Airbnb, Zillow
+
+See `examples/tools/` for more examples.
+
 ---
 
 ## 🛠️ Management APIs
@@ -182,6 +230,14 @@ monitor = client.unlimited.get_server_monitor(
 
 ---
 
+## 🧪 Development & Testing
+
+- **Full env reference**: Copy [.env.example](.env.example) to `.env` and fill in credentials.
+- **Unit tests** (no network): `pytest` or `python -m coverage run -m pytest -p no:cov tests && python -m coverage report -m`
+- **Integration tests** (live API/proxy): Set `THORDATA_INTEGRATION=true` in `.env`; optional `THORDATA_UPSTREAM_PROXY` (e.g. Clash) if behind a firewall. See [CONTRIBUTING.md](CONTRIBUTING.md#-testing-guidelines).
+
+---
+
 ## 📄 License
 
 MIT License. See [LICENSE](LICENSE) for details.

{thordata_sdk-1.5.0 → thordata_sdk-1.6.0}/pyproject.toml

@@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "thordata-sdk"
-version = "1.5.0"
+version = "1.6.0"
 description = "The Official Python SDK for Thordata - AI Data Infrastructure & Proxy Network."
 readme = "README.md"
 requires-python = ">=3.9"
@@ -120,11 +120,11 @@ ignore_missing_imports = true
 module = ["aiohttp.*", "requests.*"]
 ignore_missing_imports = true
 
-# Pytest setup
+# Pytest setup (coverage is run separately via coverage CLI for reliability)
 [tool.pytest.ini_options]
 testpaths = ["tests"]
 asyncio_mode = "auto"
-addopts = "-v --cov=thordata --cov-report=term-missing"
+addopts = "-v"
 markers = ["integration: live tests that require real credentials"]
 
 # Coverage setup

{thordata_sdk-1.5.0 → thordata_sdk-1.6.0}/src/thordata/__init__.py

@@ -5,7 +5,7 @@ Official Python client for Thordata's Proxy Network, SERP API,
 Universal Scraping API (Web Unlocker), and Web Scraper API.
 """
 
-__version__ = "1.5.0"
+__version__ = "1.6.0"
 __author__ = "Thordata Developer Team/Kael Odin"
 __email__ = "support@thordata.com"
 

{thordata_sdk-1.5.0 → thordata_sdk-1.6.0}/src/thordata/async_client.py

@@ -124,10 +124,10 @@ class AsyncThordataClient:
         ).rstrip("/")
 
         self._gateway_base_url = os.getenv(
-            "THORDATA_GATEWAY_BASE_URL", "https://api.thordata.com/api/gateway"
+            "THORDATA_GATEWAY_BASE_URL", "https://openapi.thordata.com/api/gateway"
         )
         self._child_base_url = os.getenv(
-            "THORDATA_CHILD_BASE_URL", "https://api.thordata.com/api/child"
+            "THORDATA_CHILD_BASE_URL", "https://openapi.thordata.com/api/child"
         )
 
         # URL Construction
@@ -145,7 +145,7 @@ class AsyncThordataClient:
         self._proxy_users_url = f"{shared_api_base}/proxy-users"
 
         whitelist_base = os.getenv(
-            "THORDATA_WHITELIST_BASE_URL", "https://api.thordata.com/api"
+            "THORDATA_WHITELIST_BASE_URL", "https://openapi.thordata.com/api"
         )
         self._whitelist_url = f"{whitelist_base}/whitelisted-ips"
 
@@ -352,7 +352,7 @@ class AsyncThordataClient:
         file_name: str,
         spider_id: str,
         spider_name: str,
-        parameters: dict[str, Any],
+        parameters: dict[str, Any] | list[dict[str, Any]],
         universal_params: dict[str, Any] | None = None,
     ) -> str:
         config = ScraperTaskConfig(
@@ -434,7 +434,7 @@ class AsyncThordataClient:
         file_name: str,
         spider_id: str,
         spider_name: str,
-        parameters: dict[str, Any],
+        parameters: dict[str, Any] | list[dict[str, Any]],
         common_settings: CommonSettings,
     ) -> str:
         config = VideoTaskConfig(
@@ -550,7 +550,7 @@ class AsyncThordataClient:
         file_name: str,
         spider_id: str,
         spider_name: str,
-        parameters: dict[str, Any],
+        parameters: dict[str, Any] | list[dict[str, Any]],
         universal_params: dict[str, Any] | None = None,
         *,
         max_wait: float = 600.0,
@@ -971,7 +971,12 @@ class AsyncThordataClient:
         if port:
             params["port"] = str(port)
 
-        username = os.getenv("THORDATA_RESIDENTIAL_USERNAME")
+        if product == "unlimited":
+            username = os.getenv("THORDATA_UNLIMITED_USERNAME") or os.getenv(
+                "THORDATA_RESIDENTIAL_USERNAME"
+            )
+        else:
+            username = os.getenv("THORDATA_RESIDENTIAL_USERNAME")
         if username:
             params["td-customer"] = username
 

{thordata_sdk-1.5.0 → thordata_sdk-1.6.0}/src/thordata/client.py

@@ -159,10 +159,10 @@ class ThordataClient:
         ).rstrip("/")
 
         self._gateway_base_url = os.getenv(
-            "THORDATA_GATEWAY_BASE_URL", "https://api.thordata.com/api/gateway"
+            "THORDATA_GATEWAY_BASE_URL", "https://openapi.thordata.com/api/gateway"
         )
         self._child_base_url = os.getenv(
-            "THORDATA_CHILD_BASE_URL", "https://api.thordata.com/api/child"
+            "THORDATA_CHILD_BASE_URL", "https://openapi.thordata.com/api/child"
         )
 
         # URL Construction
@@ -183,7 +183,7 @@ class ThordataClient:
         self._proxy_users_url = f"{shared_api_base}/proxy-users"
 
         whitelist_base = os.getenv(
-            "THORDATA_WHITELIST_BASE_URL", "https://api.thordata.com/api"
+            "THORDATA_WHITELIST_BASE_URL", "https://openapi.thordata.com/api"
         )
         self._whitelist_url = f"{whitelist_base}/whitelisted-ips"
 
@@ -405,7 +405,7 @@ class ThordataClient:
         file_name: str,
         spider_id: str,
         spider_name: str,
-        parameters: dict[str, Any],
+        parameters: dict[str, Any] | list[dict[str, Any]],
         universal_params: dict[str, Any] | None = None,
     ) -> str:
         config = ScraperTaskConfig(
@@ -490,7 +490,7 @@ class ThordataClient:
         file_name: str,
         spider_id: str,
         spider_name: str,
-        parameters: dict[str, Any],
+        parameters: dict[str, Any] | list[dict[str, Any]],
         common_settings: CommonSettings,
     ) -> str:
         config = VideoTaskConfig(
@@ -639,7 +639,7 @@ class ThordataClient:
         file_name: str,
         spider_id: str,
         spider_name: str,
-        parameters: dict[str, Any],
+        parameters: dict[str, Any] | list[dict[str, Any]],
         universal_params: dict[str, Any] | None = None,
         *,
         max_wait: float = 600.0,
@@ -862,7 +862,12 @@ class ThordataClient:
         if port:
             params["port"] = str(port)
 
-        username = os.getenv("THORDATA_RESIDENTIAL_USERNAME")
+        if product == "unlimited":
+            username = os.getenv("THORDATA_UNLIMITED_USERNAME") or os.getenv(
+                "THORDATA_RESIDENTIAL_USERNAME"
+            )
+        else:
+            username = os.getenv("THORDATA_RESIDENTIAL_USERNAME")
         if username:
             params["td-customer"] = username
 

{thordata_sdk-1.5.0 → thordata_sdk-1.6.0}/src/thordata/enums.py

@@ -1,6 +1,6 @@
 """
 Enumerations for the Thordata Python SDK.
-Moved to thordata.types in v1.5.0.
+Moved to thordata.types in v1.6.0.
 This file is kept for backward compatibility.
 """
 
@@ -21,7 +21,7 @@ from .types import (
     SessionType,
     TaskStatus,
     TimeRange,
-    normalize_enum_value,  # 新增
+    normalize_enum_value,
 )
 
 __all__ = [

{thordata_sdk-1.5.0 → thordata_sdk-1.6.0}/src/thordata/exceptions.py

@@ -15,6 +15,7 @@ Exception Hierarchy:
 
 from __future__ import annotations
 
+from collections.abc import Mapping
 from typing import Any
 
 # =============================================================================
@@ -235,6 +236,46 @@ class ThordataNotCollectedError(ThordataAPIError):
 # =============================================================================
 
 
+def _extract_request_id(payload: Any) -> str | None:
+    if isinstance(payload, Mapping):
+        for key in ("request_id", "requestId", "x_request_id", "x-request-id"):
+            val = payload.get(key)
+            if val is not None:
+                return str(val)
+    return None
+
+
+def _extract_retry_after(payload: Any) -> int | None:
+    if isinstance(payload, Mapping):
+        for key in ("retry_after", "retryAfter", "retry-after"):
+            val = payload.get(key)
+            if isinstance(val, int):
+                return val
+            if isinstance(val, str) and val.isdigit():
+                return int(val)
+    return None
+
+
+def _build_error_message(
+    message: str,
+    *,
+    status_code: int | None,
+    code: int | None,
+    request_id: str | None,
+) -> str:
+    parts: list[str] = [message]
+    meta: list[str] = []
+    if status_code is not None:
+        meta.append(f"http={status_code}")
+    if code is not None and code != status_code:
+        meta.append(f"code={code}")
+    if request_id:
+        meta.append(f"request_id={request_id}")
+    if meta:
+        parts.append("(" + ", ".join(meta) + ")")
+    return " ".join(parts)
+
+
 def raise_for_code(
     message: str,
     *,
@@ -266,49 +307,59 @@ def raise_for_code(
     # Determine the effective error code.
     # Prefer payload `code` when present and not success (200),
     # otherwise fall back to HTTP status when it indicates an error.
+    # Determine the effective error code for routing.
     effective_code: int | None = None
-
     if code is not None and code != 200:
         effective_code = code
-    elif status_code is not None and status_code != 200:
+    elif status_code is not None and status_code >= 400:
         effective_code = status_code
     else:
         effective_code = code if code is not None else status_code
 
+    # Extract additional context from payload
+    final_request_id = request_id or _extract_request_id(payload)
+
+    # Build a consistent, informative error message
+    final_message = _build_error_message(
+        message,
+        status_code=status_code,
+        code=code,
+        request_id=final_request_id,
+    )
+
+    # Prepare common arguments for exception constructors
     kwargs = {
         "status_code": status_code,
         "code": code,
         "payload": payload,
-        "request_id": request_id,
+        "request_id": final_request_id,
     }
 
+    # --- Route to the correct exception class ---
+
     # Not collected (API payload code 300, often retryable, not billed)
-    # Check this FIRST since 300 is in API_CODES, not HTTP_STATUS_CODES
     if effective_code in ThordataNotCollectedError.API_CODES:
-        raise ThordataNotCollectedError(message, **kwargs)
+        raise ThordataNotCollectedError(final_message, **kwargs)
 
-    # Auth errors
+    # Auth errors (401, 403)
     if effective_code in ThordataAuthError.HTTP_STATUS_CODES:
-        raise ThordataAuthError(message, **kwargs)
+        raise ThordataAuthError(final_message, **kwargs)
 
-    # Rate limit errors
+    # Rate limit errors (429, 402)
     if effective_code in ThordataRateLimitError.HTTP_STATUS_CODES:
-        # Try to extract retry_after from payload
-        retry_after = None
-        if isinstance(payload, dict):
-            retry_after = payload.get("retry_after")
-        raise ThordataRateLimitError(message, retry_after=retry_after, **kwargs)
+        retry_after = _extract_retry_after(payload)
+        raise ThordataRateLimitError(final_message, retry_after=retry_after, **kwargs)
 
-    # Server errors
+    # Server errors (5xx)
     if effective_code is not None and 500 <= effective_code < 600:
-        raise ThordataServerError(message, **kwargs)
+        raise ThordataServerError(final_message, **kwargs)
 
-    # Validation errors
+    # Validation errors (400, 422)
     if effective_code in ThordataValidationError.HTTP_STATUS_CODES:
-        raise ThordataValidationError(message, **kwargs)
+        raise ThordataValidationError(final_message, **kwargs)
 
-    # Generic API error
-    raise ThordataAPIError(message, **kwargs)
+    # Fallback to generic API error if no specific match
+    raise ThordataAPIError(final_message, **kwargs)
 
 
 # =============================================================================

{thordata_sdk-1.5.0 → thordata_sdk-1.6.0}/src/thordata/models.py

@@ -1,6 +1,6 @@
 """
 Data models for the Thordata Python SDK.
-Moved to thordata.types in v1.5.0.
+Moved to thordata.types in v1.6.0.
 This file is kept for backward compatibility.
 """
 

{thordata_sdk-1.5.0 → thordata_sdk-1.6.0}/src/thordata/retry.py

@@ -186,7 +186,7 @@ def with_retry(
                     if isinstance(e, ThordataRateLimitError) and e.retry_after:
                         delay = max(delay, e.retry_after)
 
-                    logger.warning(
+                    logger.info(
                         f"Retry attempt {attempt + 1}/{config.max_retries} "
                         f"after {delay:.2f}s due to: {e}"
                     )

{thordata_sdk-1.5.0 → thordata_sdk-1.6.0}/src/thordata/tools/__init__.py

@@ -5,15 +5,19 @@ High-level abstractions for specific scraping targets.
 
 from .base import ToolRequest, VideoToolRequest
 from .code import GitHub
-from .ecommerce import Amazon
+from .ecommerce import Amazon, Walmart, eBay
+from .professional import Crunchbase, Glassdoor, Indeed
 from .search import GoogleMaps, GooglePlay, GoogleShopping
 from .social import Facebook, Instagram, LinkedIn, Reddit, TikTok, Twitter
+from .travel import Airbnb, Booking, Zillow
 from .video import YouTube
 
 __all__ = [
     "ToolRequest",
     "VideoToolRequest",
     "Amazon",
+    "eBay",
+    "Walmart",
     "GoogleMaps",
     "GoogleShopping",
     "GooglePlay",
@@ -25,4 +29,10 @@ __all__ = [
     "Reddit",
     "YouTube",
     "GitHub",
+    "Indeed",
+    "Glassdoor",
+    "Crunchbase",
+    "Booking",
+    "Zillow",
+    "Airbnb",
 ]

thordata_sdk-1.6.0/src/thordata/tools/code.py

@@ -0,0 +1,39 @@
+"""
+Code Repository Scraper Tools (GitHub, etc.)
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+from .base import ToolRequest
+
+
+class GitHub:
+    """Namespace for GitHub tools."""
+
+    @dataclass
+    class Repository(ToolRequest):
+        """Github Repository Scraper by Repo URL"""
+
+        SPIDER_ID = "github_repository_by-repo-url"
+        SPIDER_NAME = "github.com"
+        repo_url: str
+
+    @dataclass
+    class RepositoryBySearchUrl(ToolRequest):
+        """Github Repository Scraper by Search URL"""
+
+        SPIDER_ID = "github_repository_by-search-url"
+        SPIDER_NAME = "github.com"
+        search_url: str
+        page_turning: int | None = None
+        max_num: int | None = None
+
+    @dataclass
+    class RepositoryByUrl(ToolRequest):
+        """Github Repository Scraper by URL"""
+
+        SPIDER_ID = "github_repository_by-url"
+        SPIDER_NAME = "github.com"
+        url: str