thordata-sdk 0.3.1__tar.gz → 0.4.0__tar.gz

thordata_sdk-0.4.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Thordata · AI Proxy & Web Data
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

thordata_sdk-0.4.0/PKG-INFO

@@ -0,0 +1,678 @@
+Metadata-Version: 2.4
+Name: thordata-sdk
+Version: 0.4.0
+Summary: The Official Python SDK for Thordata - AI Data Infrastructure & Proxy Network.
+Author-email: Thordata Developer Team <support@thordata.com>
+License: MIT
+Project-URL: Homepage, https://www.thordata.com
+Project-URL: Documentation, https://github.com/Thordata/thordata-python-sdk#readme
+Project-URL: Source, https://github.com/Thordata/thordata-python-sdk
+Project-URL: Tracker, https://github.com/Thordata/thordata-python-sdk/issues
+Project-URL: Changelog, https://github.com/Thordata/thordata-python-sdk/blob/main/CHANGELOG.md
+Keywords: web scraping,proxy,residential proxy,datacenter proxy,ai,llm,data-mining,serp,thordata,web scraper,anti-bot bypass
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Topic :: Internet :: Proxy Servers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Typing :: Typed
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests>=2.25.0
+Requires-Dist: aiohttp>=3.8.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
+Requires-Dist: black>=23.0.0; extra == "dev"
+Requires-Dist: ruff>=0.1.0; extra == "dev"
+Requires-Dist: mypy>=1.0.0; extra == "dev"
+Requires-Dist: types-requests>=2.28.0; extra == "dev"
+Dynamic: license-file
+
+# Thordata Python SDK
+
+<div align="center">
+
+**Official Python client for Thordata's Proxy Network, SERP API, Web Unlocker, and Web Scraper API.**
+
+*Async-ready, type-safe, built for AI agents and large-scale data collection.*
+
+[![CI](https://github.com/Thordata/thordata-python-sdk/actions/workflows/ci.yml/badge.svg)](https://github.com/Thordata/thordata-python-sdk/actions/workflows/ci.yml)
+[![PyPI version](https://img.shields.io/pypi/v/thordata-sdk?color=blue)](https://pypi.org/project/thordata-sdk/)
+[![Python](https://img.shields.io/badge/python-3.8+-blue)](https://python.org)
+[![License](https://img.shields.io/badge/license-MIT-green)](LICENSE)
+[![Typed](https://img.shields.io/badge/typing-typed-purple)](https://github.com/Thordata/thordata-python-sdk)
+
+[Documentation](https://doc.thordata.com) • [Dashboard](https://www.thordata.com) • [Examples](examples/) • [Changelog](CHANGELOG.md)
+
+</div>
+
+---
+
+## ✨ Features
+
+| Feature | Description |
+|---------|-------------|
+| 🌐 **Proxy Network** | Residential, Mobile, Datacenter, ISP proxies with geo-targeting |
+| 🔍 **SERP API** | Google, Bing, Yandex, DuckDuckGo, Baidu search results |
+| 🔓 **Web Unlocker** | Bypass Cloudflare, CAPTCHAs, anti-bot systems automatically |
+| 🕷️ **Web Scraper** | Async task-based scraping for complex sites |
+| ⚡ **Async Support** | Full async/await support with aiohttp |
+| 🔄 **Auto Retry** | Configurable retry with exponential backoff |
+| 📝 **Type Safe** | Full type annotations for IDE autocomplete |
+
+---
+
+## 📦 Installation
+
+```bash
+pip install thordata-sdk
+```
+
+For development:
+
+```bash
+pip install thordata-sdk[dev]
+```
+
+---
+
+## 🚀 Quick Start
+
+### Get Your Credentials
+
+1. Sign up at [thordata.com](https://www.thordata.com)
+2. Navigate to your Dashboard
+3. Copy your Scraper Token, Public Token, and Public Key
+
+### Basic Usage
+
+```python
+from thordata import ThordataClient
+
+# Initialize the client
+client = ThordataClient(
+    scraper_token="your_scraper_token",
+    public_token="your_public_token",  # Optional, for task APIs
+    public_key="your_public_key"        # Optional, for task APIs
+)
+
+# Make a request through the proxy network
+response = client.get("https://httpbin.org/ip")
+print(response.json())
+# {'origin': '123.45.67.89'}  # Residential IP
+```
+
+### Environment Variables
+
+Create a `.env` file:
+
+```env
+THORDATA_SCRAPER_TOKEN=your_scraper_token
+THORDATA_PUBLIC_TOKEN=your_public_token
+THORDATA_PUBLIC_KEY=your_public_key
+```
+
+Then use with python-dotenv:
+
+```python
+import os
+from dotenv import load_dotenv
+from thordata import ThordataClient
+
+load_dotenv()
+
+client = ThordataClient(
+    scraper_token=os.getenv("THORDATA_SCRAPER_TOKEN"),
+    public_token=os.getenv("THORDATA_PUBLIC_TOKEN"),
+    public_key=os.getenv("THORDATA_PUBLIC_KEY"),
+)
+```
+
+---
+
+## 📖 Usage Guide
+
+### 1. Proxy Network
+
+#### Basic Proxy Request
+
+```python
+from thordata import ThordataClient
+
+client = ThordataClient(scraper_token="your_token")
+
+# GET request through proxy
+response = client.get("https://example.com")
+print(response.text)
+
+# POST request through proxy
+response = client.post("https://httpbin.org/post", json={"key": "value"})
+print(response.json())
+```
+
+#### Geo-Targeting
+
+```python
+from thordata import ThordataClient, ProxyConfig
+
+client = ThordataClient(scraper_token="your_token")
+
+# Create a proxy config with geo-targeting
+config = ProxyConfig(
+    username="your_username",
+    password="your_password",
+    country="us",           # Target country
+    state="california",     # Target state
+    city="los_angeles",     # Target city
+)
+
+response = client.get("https://httpbin.org/ip", proxy_config=config)
+print(response.json())
+```
+
+#### Sticky Sessions
+
+Keep the same IP for multiple requests:
+
+```python
+from thordata import ThordataClient, StickySession
+
+client = ThordataClient(scraper_token="your_token")
+
+# Create a sticky session (same IP for 10 minutes)
+session = StickySession(
+    username="your_username",
+    password="your_password",
+    country="gb",
+    duration_minutes=10,
+)
+
+# All requests use the same IP
+for i in range(5):
+    response = client.get("https://httpbin.org/ip", proxy_config=session)
+    print(f"Request {i+1}: {response.json()['origin']}")
+```
+
+#### Different Proxy Products
+
+```python
+from thordata import ProxyConfig, ProxyProduct
+
+# Residential proxy (default, port 9999)
+residential = ProxyConfig(
+    username="user", password="pass",
+    product=ProxyProduct.RESIDENTIAL
+)
+
+# Mobile proxy (port 5555)
+mobile = ProxyConfig(
+    username="user", password="pass",
+    product=ProxyProduct.MOBILE
+)
+
+# Datacenter proxy (port 7777)
+datacenter = ProxyConfig(
+    username="user", password="pass",
+    product=ProxyProduct.DATACENTER
+)
+```
+
+### 2. SERP API (Search Engine Results)
+
+#### Basic Search
+
+```python
+from thordata import ThordataClient, Engine
+
+client = ThordataClient(scraper_token="your_token")
+
+# Google search
+results = client.serp_search(
+    query="python programming",
+    engine=Engine.GOOGLE,
+    num=10
+)
+
+# Print organic results
+for result in results.get("organic", []):
+    print(f"{result['title']}: {result['link']}")
+```
+
+#### Advanced Search Options
+
+```python
+from thordata import ThordataClient, SerpRequest
+
+client = ThordataClient(scraper_token="your_token")
+
+# Create a detailed search request
+request = SerpRequest(
+    query="best laptops 2024",
+    engine="google",
+    num=20,
+    country="us",
+    language="en",
+    search_type="shopping",     # shopping, news, images, videos
+    time_filter="month",        # hour, day, week, month, year
+    safe_search=True,
+    device="mobile",            # desktop, mobile, tablet
+)
+
+results = client.serp_search_advanced(request)
+```
+
+#### Multiple Search Engines
+
+```python
+from thordata import ThordataClient, Engine
+
+client = ThordataClient(scraper_token="your_token")
+
+# Google
+google_results = client.serp_search("AI news", engine=Engine.GOOGLE)
+
+# Bing
+bing_results = client.serp_search("AI news", engine=Engine.BING)
+
+# Yandex (Russian search engine)
+yandex_results = client.serp_search("AI news", engine=Engine.YANDEX)
+
+# DuckDuckGo
+ddg_results = client.serp_search("AI news", engine=Engine.DUCKDUCKGO)
+```
+
+### 3. Web Unlocker (Universal Scraping API)
+
+Automatically bypass anti-bot protections:
+
+#### Basic Usage
+
+```python
+from thordata import ThordataClient
+
+client = ThordataClient(scraper_token="your_token")
+
+# Get HTML content
+html = client.universal_scrape(
+    url="https://example.com",
+    js_render=True,  # Enable JavaScript rendering
+)
+print(html[:500])
+```
+
+#### Advanced Options
+
+```python
+from thordata import ThordataClient, UniversalScrapeRequest
+
+client = ThordataClient(scraper_token="your_token")
+
+request = UniversalScrapeRequest(
+    url="https://example.com",
+    js_render=True,
+    output_format="html",
+    country="us",
+    block_resources="image,font",  # Speed up by blocking resources
+    clean_content="js,css",        # Remove JS/CSS from output
+    wait=5000,                     # Wait 5 seconds after load
+    wait_for=".content-loaded",    # Wait for CSS selector
+    headers=[
+        {"name": "Accept-Language", "value": "en-US"}
+    ],
+    cookies=[
+        {"name": "session", "value": "abc123"}
+    ],
+)
+
+html = client.universal_scrape_advanced(request)
+```
+
+#### Take Screenshots
+
+```python
+from thordata import ThordataClient
+
+client = ThordataClient(scraper_token="your_token")
+
+# Get PNG screenshot
+png_bytes = client.universal_scrape(
+    url="https://example.com",
+    js_render=True,
+    output_format="png",
+)
+
+# Save to file
+with open("screenshot.png", "wb") as f:
+    f.write(png_bytes)
+```
+
+### 4. Web Scraper API (Async Tasks)
+
+For complex scraping jobs that run asynchronously:
+
+```python
+from thordata import ThordataClient
+
+client = ThordataClient(
+    scraper_token="your_token",
+    public_token="your_public_token",
+    public_key="your_public_key",
+)
+
+# Create a scraping task
+task_id = client.create_scraper_task(
+    file_name="youtube_channel_data",
+    spider_id="youtube_video-post_by-url",  # From Dashboard
+    spider_name="youtube.com",
+    parameters={
+        "url": "https://www.youtube.com/@PewDiePie/videos",
+        "num_of_posts": "50"
+    }
+)
+print(f"Task created: {task_id}")
+
+# Wait for completion (with timeout)
+status = client.wait_for_task(task_id, max_wait=300)
+print(f"Task status: {status}")
+
+# Get results
+if status in ("ready", "success"):
+    download_url = client.get_task_result(task_id)
+    print(f"Download: {download_url}")
+```
+
+### 5. Async Client (High Concurrency)
+
+For maximum performance with concurrent requests:
+
+```python
+import asyncio
+from thordata import AsyncThordataClient
+
+async def main():
+    async with AsyncThordataClient(
+        scraper_token="your_token",
+        public_token="your_public_token",
+        public_key="your_public_key",
+    ) as client:
+        
+        # Concurrent proxy requests
+        urls = [
+            "https://httpbin.org/ip",
+            "https://httpbin.org/headers",
+            "https://httpbin.org/user-agent",
+        ]
+        
+        tasks = [client.get(url) for url in urls]
+        responses = await asyncio.gather(*tasks)
+        
+        for resp in responses:
+            print(await resp.json())
+
+asyncio.run(main())
+```
+
+#### Async SERP Search
+
+```python
+import asyncio
+from thordata import AsyncThordataClient, Engine
+
+async def search_multiple():
+    async with AsyncThordataClient(scraper_token="your_token") as client:
+        queries = ["python", "javascript", "rust", "go"]
+        
+        tasks = [
+            client.serp_search(q, engine=Engine.GOOGLE)
+            for q in queries
+        ]
+        
+        results = await asyncio.gather(*tasks)
+        
+        for query, result in zip(queries, results):
+            count = len(result.get("organic", []))
+            print(f"{query}: {count} results")
+
+asyncio.run(search_multiple())
+```
+
+### 6. Location APIs
+
+Discover available geo-targeting options:
+
+```python
+from thordata import ThordataClient, ProxyType
+
+client = ThordataClient(
+    scraper_token="your_token",
+    public_token="your_public_token",
+    public_key="your_public_key",
+)
+
+# List all supported countries
+countries = client.list_countries(proxy_type=ProxyType.RESIDENTIAL)
+print(f"Supported countries: {len(countries)}")
+
+# List states for a country
+states = client.list_states("US")
+for state in states[:5]:
+    print(f"  {state['state_code']}: {state['state_name']}")
+
+# List cities
+cities = client.list_cities("US", state_code="california")
+print(f"Cities in California: {len(cities)}")
+
+# List ASNs (for ISP targeting)
+asns = client.list_asn("US")
+for asn in asns[:5]:
+    print(f"  {asn['asn_code']}: {asn['asn_name']}")
+```
+
+### 7. Error Handling
+
+```python
+from thordata import (
+    ThordataClient,
+    ThordataError,
+    ThordataAuthError,
+    ThordataRateLimitError,
+    ThordataNetworkError,
+    ThordataTimeoutError,
+)
+
+client = ThordataClient(scraper_token="your_token")
+
+try:
+    result = client.serp_search("test query")
+except ThordataAuthError as e:
+    print(f"Authentication failed: {e}")
+    print(f"Check your token. Status code: {e.status_code}")
+except ThordataRateLimitError as e:
+    print(f"Rate limited: {e}")
+    if e.retry_after:
+        print(f"Retry after {e.retry_after} seconds")
+except ThordataTimeoutError as e:
+    print(f"Request timed out: {e}")
+except ThordataNetworkError as e:
+    print(f"Network error: {e}")
+except ThordataError as e:
+    print(f"General error: {e}")
+```
+
+### 8. Retry Configuration
+
+Customize automatic retry behavior:
+
+```python
+from thordata import ThordataClient, RetryConfig
+
+# Custom retry configuration
+retry_config = RetryConfig(
+    max_retries=5,           # Maximum retry attempts
+    backoff_factor=2.0,      # Exponential backoff multiplier
+    max_backoff=120.0,       # Maximum wait between retries
+    jitter=True,             # Add randomness to prevent thundering herd
+)
+
+client = ThordataClient(
+    scraper_token="your_token",
+    retry_config=retry_config,
+)
+
+# Requests will automatically retry on transient failures
+response = client.get("https://example.com")
+```
+
+---
+
+## 🔧 Configuration Reference
+
+### ThordataClient Parameters
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| scraper_token | str | required | API token from Dashboard |
+| public_token | str | None | Public API token (for tasks/locations) |
+| public_key | str | None | Public API key |
+| proxy_host | str | "pr.thordata.net" | Proxy gateway host |
+| proxy_port | int | 9999 | Proxy gateway port |
+| timeout | int | 30 | Default request timeout (seconds) |
+| retry_config | RetryConfig | None | Retry configuration |
+
+### ProxyConfig Parameters
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| username | str | required | Proxy username |
+| password | str | required | Proxy password |
+| product | ProxyProduct | RESIDENTIAL | Proxy type |
+| country | str | None | ISO 3166-1 alpha-2 code |
+| state | str | None | State name (lowercase) |
+| city | str | None | City name (lowercase) |
+| continent | str | None | Continent code (af/an/as/eu/na/oc/sa) |
+| asn | str | None | ASN code (requires country) |
+| session_id | str | None | Session ID for sticky sessions |
+| session_duration | int | None | Session duration (1-90 minutes) |
+
+### Proxy Products & Ports
+
+| Product | Port | Description |
+|---------|------|-------------|
+| RESIDENTIAL | 9999 | Rotating residential IPs |
+| MOBILE | 5555 | Mobile carrier IPs |
+| DATACENTER | 7777 | Datacenter IPs |
+| ISP | 6666 | Static ISP IPs |
+
+---
+
+## 📁 Project Structure
+
+```
+thordata-python-sdk/
+├── src/thordata/
+│   ├── __init__.py         # Public API exports
+│   ├── client.py           # Sync client
+│   ├── async_client.py     # Async client
+│   ├── models.py           # Data models (ProxyConfig, SerpRequest, etc.)
+│   ├── enums.py            # Enumerations
+│   ├── exceptions.py       # Exception hierarchy
+│   ├── retry.py            # Retry mechanism
+│   └── _utils.py           # Internal utilities
+├── tests/                   # Test suite
+├── examples/                # Usage examples
+├── pyproject.toml          # Package configuration
+└── README.md
+```
+
+---
+
+## 🧪 Development
+
+### Setup
+
+```bash
+# Clone the repository
+git clone https://github.com/Thordata/thordata-python-sdk.git
+cd thordata-python-sdk
+
+# Create virtual environment
+python -m venv venv
+source venv/bin/activate  # On Windows: venv\Scripts\activate
+
+# Install with dev dependencies
+pip install -e ".[dev]"
+```
+
+### Run Tests
+
+```bash
+# Run all tests
+pytest
+
+# Run with coverage
+pytest --cov=thordata --cov-report=html
+
+# Run specific test file
+pytest tests/test_client.py -v
+```
+
+### Code Quality
+
+```bash
+# Format code
+black src tests
+
+# Lint
+ruff check src tests
+
+# Type check
+mypy src
+```
+
+---
+
+## 📝 Changelog
+
+See [CHANGELOG.md](CHANGELOG.md) for version history.
+
+---
+
+## 🤝 Contributing
+
+Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
+
+1. Fork the repository
+2. Create a feature branch (`git checkout -b feature/amazing-feature`)
+3. Commit your changes (`git commit -m 'Add amazing feature'`)
+4. Push to the branch (`git push origin feature/amazing-feature`)
+5. Open a Pull Request
+
+---
+
+## 📄 License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+---
+
+## 🆘 Support
+
+- 📧 **Email**: support@thordata.com
+- 📚 **Documentation**: [doc.thordata.com](https://doc.thordata.com)
+- 🐛 **Issues**: [GitHub Issues](https://github.com/Thordata/thordata-python-sdk/issues)
+- 💬 **Dashboard**: [thordata.com](https://www.thordata.com)
+
+<div align="center">
+<sub>Built with ❤️ by the Thordata Team</sub>
+</div>