thordata-sdk 1.4.0__tar.gz → 1.6.0__tar.gz

thordata_sdk-1.6.0/PKG-INFO

@@ -0,0 +1,287 @@
+Metadata-Version: 2.4
+Name: thordata-sdk
+Version: 1.6.0
+Summary: The Official Python SDK for Thordata - AI Data Infrastructure & Proxy Network.
+Author-email: Thordata Developer Team <support@thordata.com>
+License: MIT
+Project-URL: Homepage, https://www.thordata.com
+Project-URL: Documentation, https://github.com/Thordata/thordata-python-sdk#readme
+Project-URL: Source, https://github.com/Thordata/thordata-python-sdk
+Project-URL: Tracker, https://github.com/Thordata/thordata-python-sdk/issues
+Project-URL: Changelog, https://github.com/Thordata/thordata-python-sdk/blob/main/CHANGELOG.md
+Keywords: web scraping,proxy,residential proxy,datacenter proxy,ai,llm,data-mining,serp,thordata,web scraper,anti-bot bypass
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Topic :: Internet :: Proxy Servers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Typing :: Typed
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests>=2.25.0
+Requires-Dist: aiohttp>=3.9.0
+Requires-Dist: PySocks>=1.7.1
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
+Requires-Dist: pytest-httpserver>=1.0.0; extra == "dev"
+Requires-Dist: python-dotenv>=1.0.0; extra == "dev"
+Requires-Dist: black>=25.11.0; extra == "dev"
+Requires-Dist: ruff>=0.1.0; extra == "dev"
+Requires-Dist: mypy>=1.0.0; extra == "dev"
+Requires-Dist: types-requests>=2.28.0; extra == "dev"
+Requires-Dist: aioresponses>=0.7.6; extra == "dev"
+Dynamic: license-file
+
+# Thordata Python SDK
+
+<div align="center">
+
+<img src="https://img.shields.io/badge/Thordata-AI%20Infrastructure-blue?style=for-the-badge" alt="Thordata Logo">
+
+**The Official Python Client for Thordata APIs**
+
+*Proxy Network • SERP API • Web Unlocker • Web Scraper API*
+
+[![PyPI version](https://img.shields.io/pypi/v/thordata-sdk.svg?style=flat-square)](https://pypi.org/project/thordata-sdk/)
+[![Python Versions](https://img.shields.io/pypi/pyversions/thordata-sdk.svg?style=flat-square)](https://pypi.org/project/thordata-sdk/)
+[![License](https://img.shields.io/badge/license-MIT-green?style=flat-square)](LICENSE)
+[![CI Status](https://img.shields.io/github/actions/workflow/status/Thordata/thordata-python-sdk/ci.yml?branch=main&style=flat-square)](https://github.com/Thordata/thordata-python-sdk/actions)
+
+</div>
+
+---
+
+## 📖 Introduction
+
+The **Thordata Python SDK v1.6.0** is a production-ready wrapper for Thordata's AI data infrastructure. It is architected for high reliability, strict type safety, and maximum performance.
+
+**Why v1.6.0?**
+*   **🛡️ Bulletproof Networking**: Custom core handles `HTTP`, `HTTPS`, and `SOCKS5h` (Remote DNS) tunneling, solving common SSL/TLS handshake issues in complex network environments.
+*   **⚡ Async First**: First-class `asyncio` support with `aiohttp` for high-concurrency scraping (1000+ RPS).
+*   **🧩 100% API Coverage**: Every endpoint documented by Thordata (including Hourly Usage, Server Monitor, and Task Management) is implemented.
+*   **🤖 Type Safe**: Fully typed (`mypy` strict) for excellent IDE autocompletion and error checking.
+
+---
+
+## 📦 Installation
+
+```bash
+pip install thordata-sdk
+```
+
+---
+
+## 🔐 Configuration
+
+Set environment variables to avoid hardcoding credentials. **Full reference:** copy [.env.example](.env.example) to `.env` and fill in values.
+
+```bash
+# [Scraping APIs]
+export THORDATA_SCRAPER_TOKEN="your_scraper_token"
+
+# [Management APIs]
+export THORDATA_PUBLIC_TOKEN="your_public_token"
+export THORDATA_PUBLIC_KEY="your_public_key"
+
+# [Proxy: Residential / Unlimited / Datacenter / Mobile / ISP]
+export THORDATA_RESIDENTIAL_USERNAME="your_username"
+export THORDATA_RESIDENTIAL_PASSWORD="your_password"
+# Optional: Unlimited (high-bandwidth) if your plan has separate credentials
+# export THORDATA_UNLIMITED_USERNAME="..."
+# export THORDATA_UNLIMITED_PASSWORD="..."
+
+# Optional: Upstream proxy when behind firewall (e.g. Clash Verge port 7897)
+# export THORDATA_UPSTREAM_PROXY="http://127.0.0.1:7897"
+```
+
+Default proxy port is **9999** (residential); other products use different ports (see `.env.example`).
+
+---
+
+## 🚀 Quick Start
+
+### 1. SERP Search (Google/Bing)
+
+```python
+from thordata import ThordataClient, Engine
+
+client = ThordataClient() 
+
+# Search Google
+results = client.serp_search(
+    query="latest AI trends",
+    engine=Engine.GOOGLE,
+    num=10,
+    location="United States"
+)
+
+for item in results.get("organic", []):
+    print(f"{item['title']} - {item['link']}")
+```
+
+### 2. Universal Scrape (Web Unlocker)
+
+Automatically handles JS rendering, CAPTCHAs, and fingerprinting.
+
+```python
+html = client.universal_scrape(
+    url="https://example.com",
+    js_render=True,
+    country="us",
+    wait_for=".content-loaded"  # Smart waiting
+)
+```
+
+### 3. High-Performance Proxy Tunneling
+
+Use Thordata's residential IPs directly with `requests` (Sync) or `aiohttp` (Async). The SDK handles the complex authentication and rotation logic.
+
+```python
+from thordata import ProxyConfig, ProxyProduct
+
+# Config is optional if env vars are set
+proxy = ProxyConfig(
+    product=ProxyProduct.RESIDENTIAL,
+    country="jp",
+    session_duration=10  # Sticky IP for 10 mins
+)
+
+# The client automatically routes this through Thordata's network
+response = client.get("https://httpbin.org/ip", proxy_config=proxy)
+print(response.json())
+```
+
+---
+
+## ⚙️ Advanced Usage
+
+### Async High-Concurrency
+
+Perfect for building high-throughput AI agents.
+
+```python
+import asyncio
+from thordata import AsyncThordataClient
+
+async def main():
+    async with AsyncThordataClient() as client:
+        # Fire off 10 requests in parallel
+        tasks = [client.serp_search(f"query {i}") for i in range(10)]
+        results = await asyncio.gather(*tasks)
+        print(f"Completed {len(results)} searches")
+
+asyncio.run(main())
+```
+
+### Task Management (Batch Scraping)
+
+Handle large-scale scraping jobs asynchronously.
+
+```python
+# 1. Create a task
+task_id = client.create_scraper_task(
+    file_name="daily_scrape",
+    spider_id="universal",
+    spider_name="universal",
+    parameters={"url": "https://example.com"}
+)
+
+# 2. Poll for completion (Helper method)
+status = client.wait_for_task(task_id, max_wait=600)
+
+# 3. Download results
+if status == "finished":
+    data_url = client.get_task_result(task_id)
+    print(f"Download: {data_url}")
+```
+
+### Web Scraper Tools (120+ Pre-built Tools)
+
+Use pre-built tools for popular platforms. See [Tool Coverage Matrix](docs/TOOL_COVERAGE_MATRIX.md) for full list.
+
+```python
+from thordata import ThordataClient
+from thordata.tools import Amazon, GoogleMaps, YouTube, TikTok, eBay, Walmart
+
+client = ThordataClient()
+
+# Amazon Product by ASIN
+task_id = client.run_tool(Amazon.ProductByAsin(asin="B0BZYCJK89"))
+
+# Google Maps by Place ID
+task_id = client.run_tool(GoogleMaps.DetailsByPlaceId(place_id="ChIJPTacEpBQwokRKwIlDXelxkA"))
+
+# YouTube Video Download
+from thordata import CommonSettings
+settings = CommonSettings(resolution="<=360p", video_codec="vp9")
+task_id = client.run_tool(YouTube.VideoDownload(
+    url="https://www.youtube.com/watch?v=jNQXAC9IVRw",
+    common_settings=settings
+))
+
+# Wait and get results
+status = client.wait_for_task(task_id, max_wait=300)
+if status == "ready":
+    download_url = client.get_task_result(task_id)
+    print(f"Results: {download_url}")
+```
+
+**Available Platforms:**
+- **E-Commerce**: Amazon, eBay, Walmart
+- **Social Media**: TikTok, Instagram, Facebook, Twitter/X, Reddit, LinkedIn
+- **Search**: Google Maps, Google Shopping, Google Play
+- **Video**: YouTube (download, info, subtitles)
+- **Code**: GitHub
+- **Professional**: Indeed, Glassdoor, Crunchbase
+- **Travel/Real Estate**: Booking, Airbnb, Zillow
+
+See `examples/tools/` for more examples.
+
+---
+
+## 🛠️ Management APIs
+
+Manage your infrastructure programmatically.
+
+```python
+# Check Balance
+balance = client.get_traffic_balance()
+
+# Manage Whitelist
+client.add_whitelist_ip("1.2.3.4")
+
+# Create Sub-users
+client.create_proxy_user("new_user", "pass123", traffic_limit=500)
+
+# Monitor Unlimited Proxies
+monitor = client.unlimited.get_server_monitor(
+    ins_id="ins-123", 
+    region="us", 
+    start_time=1700000000, 
+    end_time=1700003600
+)
+```
+
+---
+
+## 🧪 Development & Testing
+
+- **Full env reference**: Copy [.env.example](.env.example) to `.env` and fill in credentials.
+- **Unit tests** (no network): `pytest` or `python -m coverage run -m pytest -p no:cov tests && python -m coverage report -m`
+- **Integration tests** (live API/proxy): Set `THORDATA_INTEGRATION=true` in `.env`; optional `THORDATA_UPSTREAM_PROXY` (e.g. Clash) if behind a firewall. See [CONTRIBUTING.md](CONTRIBUTING.md#-testing-guidelines).
+
+---
+
+## 📄 License
+
+MIT License. See [LICENSE](LICENSE) for details.

thordata_sdk-1.6.0/README.md

@@ -0,0 +1,243 @@
+# Thordata Python SDK
+
+<div align="center">
+
+<img src="https://img.shields.io/badge/Thordata-AI%20Infrastructure-blue?style=for-the-badge" alt="Thordata Logo">
+
+**The Official Python Client for Thordata APIs**
+
+*Proxy Network • SERP API • Web Unlocker • Web Scraper API*
+
+[![PyPI version](https://img.shields.io/pypi/v/thordata-sdk.svg?style=flat-square)](https://pypi.org/project/thordata-sdk/)
+[![Python Versions](https://img.shields.io/pypi/pyversions/thordata-sdk.svg?style=flat-square)](https://pypi.org/project/thordata-sdk/)
+[![License](https://img.shields.io/badge/license-MIT-green?style=flat-square)](LICENSE)
+[![CI Status](https://img.shields.io/github/actions/workflow/status/Thordata/thordata-python-sdk/ci.yml?branch=main&style=flat-square)](https://github.com/Thordata/thordata-python-sdk/actions)
+
+</div>
+
+---
+
+## 📖 Introduction
+
+The **Thordata Python SDK v1.6.0** is a production-ready wrapper for Thordata's AI data infrastructure. It is architected for high reliability, strict type safety, and maximum performance.
+
+**Why v1.6.0?**
+*   **🛡️ Bulletproof Networking**: Custom core handles `HTTP`, `HTTPS`, and `SOCKS5h` (Remote DNS) tunneling, solving common SSL/TLS handshake issues in complex network environments.
+*   **⚡ Async First**: First-class `asyncio` support with `aiohttp` for high-concurrency scraping (1000+ RPS).
+*   **🧩 100% API Coverage**: Every endpoint documented by Thordata (including Hourly Usage, Server Monitor, and Task Management) is implemented.
+*   **🤖 Type Safe**: Fully typed (`mypy` strict) for excellent IDE autocompletion and error checking.
+
+---
+
+## 📦 Installation
+
+```bash
+pip install thordata-sdk
+```
+
+---
+
+## 🔐 Configuration
+
+Set environment variables to avoid hardcoding credentials. **Full reference:** copy [.env.example](.env.example) to `.env` and fill in values.
+
+```bash
+# [Scraping APIs]
+export THORDATA_SCRAPER_TOKEN="your_scraper_token"
+
+# [Management APIs]
+export THORDATA_PUBLIC_TOKEN="your_public_token"
+export THORDATA_PUBLIC_KEY="your_public_key"
+
+# [Proxy: Residential / Unlimited / Datacenter / Mobile / ISP]
+export THORDATA_RESIDENTIAL_USERNAME="your_username"
+export THORDATA_RESIDENTIAL_PASSWORD="your_password"
+# Optional: Unlimited (high-bandwidth) if your plan has separate credentials
+# export THORDATA_UNLIMITED_USERNAME="..."
+# export THORDATA_UNLIMITED_PASSWORD="..."
+
+# Optional: Upstream proxy when behind firewall (e.g. Clash Verge port 7897)
+# export THORDATA_UPSTREAM_PROXY="http://127.0.0.1:7897"
+```
+
+Default proxy port is **9999** (residential); other products use different ports (see `.env.example`).
+
+---
+
+## 🚀 Quick Start
+
+### 1. SERP Search (Google/Bing)
+
+```python
+from thordata import ThordataClient, Engine
+
+client = ThordataClient() 
+
+# Search Google
+results = client.serp_search(
+    query="latest AI trends",
+    engine=Engine.GOOGLE,
+    num=10,
+    location="United States"
+)
+
+for item in results.get("organic", []):
+    print(f"{item['title']} - {item['link']}")
+```
+
+### 2. Universal Scrape (Web Unlocker)
+
+Automatically handles JS rendering, CAPTCHAs, and fingerprinting.
+
+```python
+html = client.universal_scrape(
+    url="https://example.com",
+    js_render=True,
+    country="us",
+    wait_for=".content-loaded"  # Smart waiting
+)
+```
+
+### 3. High-Performance Proxy Tunneling
+
+Use Thordata's residential IPs directly with `requests` (Sync) or `aiohttp` (Async). The SDK handles the complex authentication and rotation logic.
+
+```python
+from thordata import ProxyConfig, ProxyProduct
+
+# Config is optional if env vars are set
+proxy = ProxyConfig(
+    product=ProxyProduct.RESIDENTIAL,
+    country="jp",
+    session_duration=10  # Sticky IP for 10 mins
+)
+
+# The client automatically routes this through Thordata's network
+response = client.get("https://httpbin.org/ip", proxy_config=proxy)
+print(response.json())
+```
+
+---
+
+## ⚙️ Advanced Usage
+
+### Async High-Concurrency
+
+Perfect for building high-throughput AI agents.
+
+```python
+import asyncio
+from thordata import AsyncThordataClient
+
+async def main():
+    async with AsyncThordataClient() as client:
+        # Fire off 10 requests in parallel
+        tasks = [client.serp_search(f"query {i}") for i in range(10)]
+        results = await asyncio.gather(*tasks)
+        print(f"Completed {len(results)} searches")
+
+asyncio.run(main())
+```
+
+### Task Management (Batch Scraping)
+
+Handle large-scale scraping jobs asynchronously.
+
+```python
+# 1. Create a task
+task_id = client.create_scraper_task(
+    file_name="daily_scrape",
+    spider_id="universal",
+    spider_name="universal",
+    parameters={"url": "https://example.com"}
+)
+
+# 2. Poll for completion (Helper method)
+status = client.wait_for_task(task_id, max_wait=600)
+
+# 3. Download results
+if status == "finished":
+    data_url = client.get_task_result(task_id)
+    print(f"Download: {data_url}")
+```
+
+### Web Scraper Tools (120+ Pre-built Tools)
+
+Use pre-built tools for popular platforms. See [Tool Coverage Matrix](docs/TOOL_COVERAGE_MATRIX.md) for full list.
+
+```python
+from thordata import ThordataClient
+from thordata.tools import Amazon, GoogleMaps, YouTube, TikTok, eBay, Walmart
+
+client = ThordataClient()
+
+# Amazon Product by ASIN
+task_id = client.run_tool(Amazon.ProductByAsin(asin="B0BZYCJK89"))
+
+# Google Maps by Place ID
+task_id = client.run_tool(GoogleMaps.DetailsByPlaceId(place_id="ChIJPTacEpBQwokRKwIlDXelxkA"))
+
+# YouTube Video Download
+from thordata import CommonSettings
+settings = CommonSettings(resolution="<=360p", video_codec="vp9")
+task_id = client.run_tool(YouTube.VideoDownload(
+    url="https://www.youtube.com/watch?v=jNQXAC9IVRw",
+    common_settings=settings
+))
+
+# Wait and get results
+status = client.wait_for_task(task_id, max_wait=300)
+if status == "ready":
+    download_url = client.get_task_result(task_id)
+    print(f"Results: {download_url}")
+```
+
+**Available Platforms:**
+- **E-Commerce**: Amazon, eBay, Walmart
+- **Social Media**: TikTok, Instagram, Facebook, Twitter/X, Reddit, LinkedIn
+- **Search**: Google Maps, Google Shopping, Google Play
+- **Video**: YouTube (download, info, subtitles)
+- **Code**: GitHub
+- **Professional**: Indeed, Glassdoor, Crunchbase
+- **Travel/Real Estate**: Booking, Airbnb, Zillow
+
+See `examples/tools/` for more examples.
+
+---
+
+## 🛠️ Management APIs
+
+Manage your infrastructure programmatically.
+
+```python
+# Check Balance
+balance = client.get_traffic_balance()
+
+# Manage Whitelist
+client.add_whitelist_ip("1.2.3.4")
+
+# Create Sub-users
+client.create_proxy_user("new_user", "pass123", traffic_limit=500)
+
+# Monitor Unlimited Proxies
+monitor = client.unlimited.get_server_monitor(
+    ins_id="ins-123", 
+    region="us", 
+    start_time=1700000000, 
+    end_time=1700003600
+)
+```
+
+---
+
+## 🧪 Development & Testing
+
+- **Full env reference**: Copy [.env.example](.env.example) to `.env` and fill in credentials.
+- **Unit tests** (no network): `pytest` or `python -m coverage run -m pytest -p no:cov tests && python -m coverage report -m`
+- **Integration tests** (live API/proxy): Set `THORDATA_INTEGRATION=true` in `.env`; optional `THORDATA_UPSTREAM_PROXY` (e.g. Clash) if behind a firewall. See [CONTRIBUTING.md](CONTRIBUTING.md#-testing-guidelines).
+
+---
+
+## 📄 License
+
+MIT License. See [LICENSE](LICENSE) for details.

{thordata_sdk-1.4.0 → thordata_sdk-1.6.0}/pyproject.toml

@@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "thordata-sdk"
-version = "1.4.0"
+version = "1.6.0"
 description = "The Official Python SDK for Thordata - AI Data Infrastructure & Proxy Network."
 readme = "README.md"
 requires-python = ">=3.9"
@@ -120,11 +120,11 @@ ignore_missing_imports = true
 module = ["aiohttp.*", "requests.*"]
 ignore_missing_imports = true
 
-# Pytest setup
+# Pytest setup (coverage is run separately via coverage CLI for reliability)
 [tool.pytest.ini_options]
 testpaths = ["tests"]
 asyncio_mode = "auto"
-addopts = "-v --cov=thordata --cov-report=term-missing"
+addopts = "-v"
 markers = ["integration: live tests that require real credentials"]
 
 # Coverage setup

{thordata_sdk-1.4.0 → thordata_sdk-1.6.0}/src/thordata/__init__.py

@@ -3,47 +3,17 @@ Thordata Python SDK
 
 Official Python client for Thordata's Proxy Network, SERP API,
 Universal Scraping API (Web Unlocker), and Web Scraper API.
-
-Basic Usage:
-    >>> from thordata import ThordataClient
-    >>>
-    >>> client = ThordataClient(
-    ...     scraper_token="your_token",
-    ...     public_token="your_public_token",
-    ...     public_key="your_public_key"
-    ... )
-    >>>
-    >>> # Proxy request
-    >>> response = client.get("https://httpbin.org/ip")
-    >>>
-    >>> # SERP search
-    >>> results = client.serp_search("python tutorial", engine="google")
-    >>>
-    >>> # Universal scrape
-    >>> html = client.universal_scrape("https://example.com", js_render=True)
-
-Async Usage:
-    >>> from thordata import AsyncThordataClient
-    >>> import asyncio
-    >>>
-    >>> async def main():
-    ...     async with AsyncThordataClient(
-    ...         scraper_token="your_token"
-    ...     ) as client:
-    ...         response = await client.get("https://httpbin.org/ip")
-    >>>
-    >>> asyncio.run(main())
 """
 
-__version__ = "1.4.0"
-__author__ = "Thordata Developer Team"
+__version__ = "1.6.0"
+__author__ = "Thordata Developer Team/Kael Odin"
 __email__ = "support@thordata.com"
 
 # Main clients
 from .async_client import AsyncThordataClient
 from .client import ThordataClient
 
-# Enums
+# Enums (Legacy Import Path)
 from .enums import (
     BingSearchType,
     Continent,
@@ -76,7 +46,7 @@ from .exceptions import (
     ThordataValidationError,
 )
 
-# Models
+# Models (Legacy Import Path)
 from .models import (
     CommonSettings,
     ProxyConfig,
@@ -99,12 +69,9 @@ from .retry import RetryConfig
 
 # Public API
 __all__ = [
-    # Version
     "__version__",
-    # Clients
     "ThordataClient",
     "AsyncThordataClient",
-    # Enums
     "Engine",
     "GoogleSearchType",
     "BingSearchType",
@@ -120,7 +87,6 @@ __all__ = [
     "ProxyHost",
     "ProxyPort",
     "GoogleTbm",
-    # Models
     "ProxyConfig",
     "ProxyProduct",
     "ProxyServer",
@@ -135,7 +101,6 @@ __all__ = [
     "CommonSettings",
     "VideoTaskConfig",
     "TaskStatusResponse",
-    # Exceptions
     "ThordataError",
     "ThordataConfigError",
     "ThordataNetworkError",
@@ -146,6 +111,5 @@ __all__ = [
     "ThordataServerError",
     "ThordataValidationError",
     "ThordataNotCollectedError",
-    # Retry
     "RetryConfig",
 ]