webscout 8.2.6__py3-none-any.whl → 8.2.8__py3-none-any.whl

webscout/litagent/Readme.md

@@ -0,0 +1,276 @@
+# 🔥 LitAgent - The Lit User Agent Generator
+
+LitAgent is a powerful and modern user agent generator that keeps your requests fresh and undetectable! Built with style and packed with features, it's your go-to solution for managing user agents in your web scraping projects.
+
+## 🚀 Quick Start
+
+```python
+from webscout import LitAgent
+
+# Create a LitAgent instance
+agent = LitAgent()
+
+# Get a random user agent
+ua = agent.random()
+print(ua)  # Mozilla/5.0 (Windows NT 11.0) AppleWebKit/537.36 ...
+```
+
+## 🎯 Features
+
+### Browser-Specific Agents
+
+```python
+# Get agents for specific browsers
+chrome_ua = agent.chrome()    # Latest Chrome agent
+firefox_ua = agent.firefox()  # Latest Firefox agent
+safari_ua = agent.safari()    # Latest Safari agent
+edge_ua = agent.edge()       # Latest Edge agent
+opera_ua = agent.opera()     # Latest Opera agent
+```
+
+### Device-Specific Agents
+
+```python
+# Get mobile or desktop agents
+mobile_ua = agent.mobile()    # Mobile device agent
+desktop_ua = agent.desktop()  # Desktop device agent
+
+# New - Get agents for specific device types
+tablet_ua = agent.tablet()    # Tablet device agent
+tv_ua = agent.smart_tv()      # Smart TV agent
+console_ua = agent.gaming()   # Gaming console agent
+```
+
+### OS-Specific Agents
+
+```python
+# New - Get agents for specific operating systems
+windows_ua = agent.windows()  # Windows agent
+mac_ua = agent.macos()        # macOS agent
+linux_ua = agent.linux()      # Linux agent
+android_ua = agent.android()  # Android agent
+ios_ua = agent.ios()          # iOS agent
+```
+
+### Custom Agent Generation
+
+```python
+# New - Generate custom user agents with specific attributes
+custom_ua = agent.custom(
+    browser="chrome",
+    version="91.0",
+    os="windows",
+    os_version="10",
+    device_type="desktop"
+)
+```
+
+### Keep It Fresh
+
+```python
+# Refresh your agents pool anytime
+agent.refresh()  # Generates new set of agents
+
+# New - Schedule automatic refreshes
+agent.auto_refresh(interval_minutes=30)  # Auto-refresh every 30 minutes
+```
+
+## 💫 Real-World Examples
+
+### With Requests
+
+```python
+import requests
+from webscout import LitAgent
+
+agent = LitAgent()
+
+def make_request(url):
+    headers = {
+        'User-Agent': agent.random(),
+        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
+        'Accept-Language': 'en-US,en;q=0.5',
+        'Connection': 'keep-alive',
+    }
+    return requests.get(url, headers=headers)
+
+# Make requests with different agents
+response1 = make_request('https://api.example.com')  # Random agent
+response2 = make_request('https://mobile.example.com')  # Another random agent
+```
+
+### With aiohttp
+
+```python
+import aiohttp
+import asyncio
+from webscout import LitAgent
+
+agent = LitAgent()
+
+async def fetch(url):
+    headers = {'User-Agent': agent.random()}
+    async with aiohttp.ClientSession() as session:
+        async with session.get(url, headers=headers) as response:
+            return await response.text()
+
+# Use it in your async code
+async def main():
+    urls = [
+        'https://api1.example.com',
+        'https://api2.example.com',
+        'https://api3.example.com'
+    ]
+    tasks = [fetch(url) for url in urls]
+    results = await asyncio.gather(*tasks)
+    return results
+```
+
+### With Selenium
+
+```python
+from selenium import webdriver
+from webscout import LitAgent
+
+agent = LitAgent()
+
+def create_driver():
+    options = webdriver.ChromeOptions()
+    options.add_argument(f'user-agent={agent.chrome()}')
+    return webdriver.Chrome(options=options)
+
+# Use it with Selenium
+driver = create_driver()
+driver.get('https://example.com')
+```
+
+### New - With Playwright
+
+```python
+from playwright.sync_api import sync_playwright
+from webscout import LitAgent
+
+agent = LitAgent()
+
+def browse_with_playwright():
+    with sync_playwright() as p:
+        browser_options = {
+            "user_agent": agent.chrome(),
+            "viewport": {"width": 1280, "height": 720}
+        }
+        browser = p.chromium.launch()
+        context = browser.new_context(**browser_options)
+        page = context.new_page()
+        page.goto('https://example.com')
+        # Continue with your scraping logic
+        browser.close()
+```
+
+## 🌟 Pro Tips
+
+1. **Rotate Agents**: Refresh your agents pool periodically to avoid detection
+   ```python
+   agent = LitAgent()
+   for _ in range(10):
+       response = requests.get(url, headers={'User-Agent': agent.random()})
+       if _ % 3 == 0:  # Refresh every 3 requests
+           agent.refresh()
+   ```
+
+2. **Device-Specific Scraping**: Use appropriate agents for different platforms
+   ```python
+   # Mobile site scraping
+   mobile_response = requests.get(
+       'https://m.example.com',
+       headers={'User-Agent': agent.mobile()}
+   )
+
+   # Desktop site scraping
+   desktop_response = requests.get(
+       'https://example.com',
+       headers={'User-Agent': agent.desktop()}
+   )
+   ```
+
+3. **Browser Consistency**: Stick to one browser type per session
+   ```python
+   chrome_agent = agent.chrome()
+   headers = {
+       'User-Agent': chrome_agent,
+       'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
+   }
+   # Use these headers for all requests in this session
+   ```
+
+4. **New - Browser Fingerprinting Defense**:
+   ```python
+   # Create consistent browser fingerprinting
+   fingerprint = agent.generate_fingerprint(browser="chrome")
+   
+   headers = {
+       'User-Agent': fingerprint['user_agent'],
+       'Accept-Language': fingerprint['accept_language'],
+       'Accept': fingerprint['accept'],
+       'Sec-Ch-Ua': fingerprint['sec_ch_ua'],
+       'Sec-Ch-Ua-Platform': fingerprint['platform']
+   }
+   
+   # Use this consistent set for all session requests
+   ```
+
+5. **New - Multi-threading Support**:
+   ```python
+   import concurrent.futures
+   from webscout import LitAgent
+   
+   agent = LitAgent(thread_safe=True)  # Thread-safe instance
+   
+   def fetch_url(url):
+       headers = {'User-Agent': agent.random()}
+       return requests.get(url, headers=headers).text
+   
+   urls = ['https://example1.com', 'https://example2.com', 'https://example3.com']
+   
+   with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
+       results = list(executor.map(fetch_url, urls))
+   ```
+
+## 🔧 Supported Browsers & Devices
+
+- **Browsers**: Chrome, Firefox, Safari, Edge, Opera, Brave, Vivaldi
+- **Operating Systems**: Windows, macOS, Linux, Android, iOS, Chrome OS
+- **Devices**: Mobile phones, Tablets, Desktops, Game consoles, Smart TVs, Wearables
+
+## 🎨 Why LitAgent?
+
+- 🚀 Modern and up-to-date user agents
+- 💪 Easy to use, hard to detect
+- 🔄 Fresh agents on demand
+- 📱 Device-specific agents
+- 🌐 All major browsers supported
+- ⚡ Lightweight and fast
+- 🧩 Advanced fingerprinting protection
+- 🔄 Seamless proxy integration
+- 🧵 Thread-safe operation
+- 🕰️ Automatic refresh scheduling
+
+## 📊 New - Analytics and Reporting
+
+```python
+# Get statistics on your agent usage
+stats = agent.get_stats()
+print(f"Agents generated: {stats.total_generated}")
+print(f"Most used browser: {stats.top_browser}")
+print(f"Detection avoidance rate: {stats.avoidance_rate}%")
+
+# Export your usage data
+agent.export_stats('agent_usage.json')
+```
+
+## 📋 Installation
+
+```bash
+pip install webscout
+```
+
+Made with 💖 by the HelpingAI team

webscout/scout/README.md

@@ -0,0 +1,402 @@
+# 🕵️ Scout: Next-Gen Web Parsing Library
+
+<div align="center">
+
+[![Python](https://img.shields.io/badge/Python-3.8%2B-blue)](https://www.python.org/)
+[![License](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)
+[![Maintenance](https://img.shields.io/badge/Maintained-Yes-brightgreen.svg)](https://github.com/OE-LUCIFER/Webscout)
+[![Documentation](https://img.shields.io/badge/Docs-Wiki-orange)](https://github.com/OE-LUCIFER/Webscout/wiki)
+[![PRs Welcome](https://img.shields.io/badge/PRs-Welcome-brightgreen.svg)](https://github.com/OE-LUCIFER/Webscout/pulls)
+
+</div>
+
+## 📋 Overview
+
+Scout is a powerful, flexible, and performant HTML parsing library that makes web scraping a breeze! It provides intelligent HTML/XML parsing with advanced features like web crawling, text analysis, semantic extraction, and Markdown conversion. Scout goes beyond traditional parsing libraries with its intuitive API and comprehensive feature set.
+
+<details open>
+<summary><b>Why Choose Scout?</b></summary>
+
+- **Powerful Parsing**: Multiple parser backends with intelligent markup handling
+- **Advanced Analysis**: Built-in text and web content analysis tools
+- **Concurrent Crawling**: Efficient multi-threaded web crawling
+- **Flexible API**: Intuitive interface similar to BeautifulSoup but with enhanced capabilities
+- **Format Conversion**: Convert HTML to JSON, Markdown, and more
+
+</details>
+
+## 📑 Table of Contents
+
+- [Installation](#-installation)
+- [Quick Start](#-quick-start)
+- [Features](#-features)
+- [Advanced Usage](#-advanced-usage)
+- [API Reference](#-api-reference)
+- [Dependencies](#-dependencies)
+- [Supported Python Versions](#-supported-python-versions)
+- [Contributing](#-contributing)
+- [License](#-license)
+
+## 📦 Installation
+
+```bash
+pip install webscout
+```
+
+Or install the latest version from GitHub:
+
+```bash
+pip install git+https://github.com/OE-LUCIFER/Webscout.git
+```
+
+## 🚀 Quick Start
+
+### Basic Parsing
+
+```python
+from webscout.scout import Scout
+
+# Parse HTML content
+html_content = """
+<html>
+    <body>
+        <h1>Hello, Scout!</h1>
+        <div class="content">
+            <p>Web parsing made easy.</p>
+            <a href="https://example.com">Link</a>
+        </div>
+    </body>
+</html>
+"""
+
+scout = Scout(html_content)
+
+# Find elements
+title = scout.find('h1')
+links = scout.find_all('a')
+
+# Extract text
+print(title[0].get_text())  # Output: Hello, Scout!
+print(links.attrs('href'))  # Output: ['https://example.com']
+```
+
+### Web Crawling
+
+```python
+from webscout.scout import ScoutCrawler
+
+# Crawl a website with default settings
+crawler = ScoutCrawler('https://example.com')  # Default: max_pages=50
+
+# Or customize the crawler
+crawler = ScoutCrawler(
+    'https://example.com',                      # base_url
+    max_pages=100,                              # maximum pages to crawl
+    tags_to_remove=['script', 'style', 'nav']   # tags to remove from content
+)
+
+# Start crawling
+crawled_pages = crawler.crawl()
+
+for page in crawled_pages:
+    print(f"URL: {page['url']}")
+    print(f"Title: {page['title']}")
+    print(f"Links found: {len(page['links'])}")
+    print(f"Crawl depth: {page['depth']}")
+```
+
+### Text Analysis
+
+```python
+from webscout.scout import Scout
+
+# Parse a webpage
+html = """<div><h1>Climate Change</h1><p>Email us at info@example.com or call 555-123-4567.</p>
+<p>Visit https://climate-action.org for more information.</p></div>"""
+scout = Scout(html)
+
+# Analyze text and extract entities
+analysis = scout.analyze_text()
+print(f"Word frequencies: {analysis['word_count']}")
+print(f"Entities found: {analysis['entities']}")
+```
+
+## ✨ Features
+
+### 🔍 Multiple Parser Support
+
+Scout supports multiple HTML/XML parsers, allowing you to choose the best tool for your specific needs:
+
+| Parser | Description | Best For |
+|--------|-------------|----------|
+| `html.parser` | Python's built-in parser | General-purpose parsing, no dependencies |
+| `lxml` | Fast C-based parser | Performance-critical applications |
+| `html5lib` | Highly compliant HTML5 parser | Handling malformed HTML |
+| `lxml-xml` | XML parser | XML document parsing |
+
+```python
+# Choose your parser
+scout = Scout(html_content, features='lxml')  # For speed
+scout = Scout(html_content, features='html5lib')  # For compliance
+```
+
+### 🌐 Advanced Parsing Capabilities
+
+Scout provides powerful tools for navigating and manipulating HTML/XML documents:
+
+- **Element Selection**: Find elements by tag name, attributes, CSS selectors, and more
+- **Tree Traversal**: Navigate parent-child relationships and sibling elements
+- **Content Extraction**: Extract text, attributes, and structured data
+- **Document Manipulation**: Modify, replace, or remove elements
+
+```python
+# CSS selector support
+elements = scout.select('div.content > p')
+
+# Advanced find with attribute matching
+results = scout.find_all('a', attrs={'class': 'external', 'rel': 'nofollow'})
+
+# Tree traversal
+parent = element.find_parent('div')
+siblings = element.find_next_siblings('p')
+```
+
+### 🧠 Intelligent Analysis
+
+Scout includes built-in analysis tools for extracting insights from web content:
+
+#### Text Analysis
+
+```python
+# Extract and analyze text
+text = scout.get_text()
+word_counts = scout.text_analyzer.count_words(text)
+entities = scout.text_analyzer.extract_entities(text)
+```
+
+#### Web Structure Analysis
+
+```python
+# Analyze page structure
+structure = scout.analyze_page_structure()
+print(f"Most common tags: {structure['tag_distribution']}")
+print(f"Page depth: {max(structure['depth_analysis'].keys())}")
+```
+
+#### Semantic Information Extraction
+
+```python
+# Extract semantic information
+semantics = scout.extract_semantic_info()
+print(f"Headings: {semantics['headings']}")
+print(f"Lists: {len(semantics['lists']['ul']) + len(semantics['lists']['ol'])}")
+print(f"Tables: {semantics['tables']['count']}")
+```
+
+### 🕸️ Web Crawling
+
+Scout includes a powerful concurrent web crawler for fetching and analyzing multiple pages:
+
+```python
+from webscout.scout import ScoutCrawler
+
+# Create a crawler with default settings
+crawler = ScoutCrawler('https://example.com')  # Default: max_pages=50
+
+# Or customize the crawler with specific options
+crawler = ScoutCrawler(
+    'https://example.com',                      # base_url
+    max_pages=100,                              # maximum pages to crawl
+    tags_to_remove=['script', 'style', 'nav']   # tags to remove from content
+)
+
+# Start crawling
+pages = crawler.crawl()
+
+# Process results
+for page in pages:
+    print(f"URL: {page['url']}")
+    print(f"Title: {page['title']}")
+    print(f"Links: {len(page['links'])}")
+    print(f"Depth: {page['depth']}")
+```
+
+The crawler automatically:
+- Stays within the same domain as the base URL
+- Uses concurrent requests for faster crawling
+- Removes unwanted tags (like scripts and styles) for cleaner text extraction
+- Tracks crawl depth for each page
+
+### 📄 Format Conversion
+
+Scout can convert HTML to various formats:
+
+```python
+# Convert to JSON
+json_data = scout.to_json(indent=2)
+
+# Convert to Markdown
+markdown = scout.to_markdown(heading_style='ATX')
+
+# Pretty-print HTML
+pretty_html = scout.prettify()
+```
+
+## 🔬 Advanced Usage
+
+### Working with Search Results
+
+Scout's search methods return a `ScoutSearchResult` object with powerful methods for processing results:
+
+```python
+from webscout.scout import Scout
+
+scout = Scout(html_content)
+
+# Find all paragraphs
+paragraphs = scout.find_all('p')
+
+# Extract all text from results
+all_text = paragraphs.texts(separator='\n')
+
+# Extract specific attributes
+hrefs = paragraphs.attrs('href')
+
+# Filter results with a predicate function
+important = paragraphs.filter(lambda p: 'important' in p.get('class', []))
+
+# Transform results
+word_counts = paragraphs.map(lambda p: len(p.get_text().split()))
+
+# Analyze text in results
+analysis = paragraphs.analyze_text()
+```
+
+### URL Handling and Analysis
+
+```python
+from webscout.scout import Scout
+
+scout = Scout(html_content)
+
+# Parse and analyze URLs
+links = scout.extract_links(base_url='https://example.com')
+for link in links:
+    url_components = scout.url_parse(link['href'])
+    print(f"Domain: {url_components['netloc']}")
+    print(f"Path: {url_components['path']}")
+```
+
+### Metadata Extraction
+
+```python
+from webscout.scout import Scout
+
+scout = Scout(html_content)
+
+# Extract metadata
+metadata = scout.extract_metadata()
+print(f"Title: {metadata['title']}")
+print(f"Description: {metadata['description']}")
+print(f"Open Graph: {metadata['og_metadata']}")
+print(f"Twitter Card: {metadata['twitter_metadata']}")
+```
+
+### Content Hashing and Caching
+
+```python
+from webscout.scout import Scout
+
+scout = Scout(html_content)
+
+# Generate content hash
+content_hash = scout.hash_content(method='sha256')
+
+# Use caching for expensive operations
+if not scout.cache('parsed_data'):
+    data = scout.extract_semantic_info()
+    scout.cache('parsed_data', data)
+
+cached_data = scout.cache('parsed_data')
+```
+
+## 📚 API Reference
+
+### Core Classes
+
+| Class | Description |
+|-------|-------------|
+| `Scout` | Main class for HTML parsing and traversal |
+| `ScoutCrawler` | Web crawler for fetching and parsing multiple pages |
+| `ScoutTextAnalyzer` | Text analysis utilities |
+| `ScoutWebAnalyzer` | Web page analysis utilities |
+| `ScoutSearchResult` | Enhanced search results with filtering and analysis |
+| `Tag` | Represents an HTML/XML tag |
+| `NavigableString` | Represents text within an HTML/XML document |
+
+### Key Methods
+
+#### Scout Class
+
+- `__init__(markup, features='html.parser', from_encoding=None)`: Initialize with HTML content
+- `find(name, attrs={}, recursive=True, text=None)`: Find first matching element
+- `find_all(name, attrs={}, recursive=True, text=None, limit=None)`: Find all matching elements
+- `select(selector)`: Find elements using CSS selector
+- `get_text(separator=' ', strip=False)`: Extract text from document
+- `analyze_text()`: Perform text analysis
+- `analyze_page_structure()`: Analyze document structure
+- `extract_semantic_info()`: Extract semantic information
+- `extract_links(base_url=None)`: Extract all links
+- `extract_metadata()`: Extract metadata from document
+- `to_json(indent=2)`: Convert to JSON
+- `to_markdown(heading_style='ATX')`: Convert to Markdown
+- `prettify(formatter='minimal')`: Pretty-print HTML
+
+#### ScoutCrawler Class
+
+- `__init__(base_url, max_pages=50, tags_to_remove=None)`: Initialize the crawler
+- `crawl()`: Start crawling from the base URL
+- `_crawl_page(url, depth=0)`: Crawl a single page (internal method)
+- `_is_valid_url(url)`: Check if a URL is valid (internal method)
+
+For detailed API documentation, please refer to the [documentation](https://github.com/OE-LUCIFER/Webscout/wiki).
+
+## 🔧 Dependencies
+
+- `requests`: HTTP library for making web requests
+- `lxml`: XML and HTML processing library (optional, recommended)
+- `html5lib`: Standards-compliant HTML parser (optional)
+- `markdownify`: HTML to Markdown conversion
+- `concurrent.futures`: Asynchronous execution (standard library)
+
+## 🌈 Supported Python Versions
+
+- Python 3.8+
+
+## 🤝 Contributing
+
+Contributions are welcome! Here's how you can contribute:
+
+1. Fork the repository
+2. Create a feature branch (`git checkout -b feature/amazing-feature`)
+3. Commit your changes (`git commit -m 'Add some amazing feature'`)
+4. Push to the branch (`git push origin feature/amazing-feature`)
+5. Open a Pull Request
+
+Please make sure to update tests as appropriate.
+
+## 📄 License
+
+This project is licensed under the MIT License - see the LICENSE file for details.
+
+---
+
+<div align="center">
+  <p>Made with ❤️ by the Webscout team</p>
+  <p>
+    <a href="https://github.com/OE-LUCIFER/Webscout">GitHub</a> •
+    <a href="https://github.com/OE-LUCIFER/Webscout/wiki">Documentation</a> •
+    <a href="https://github.com/OE-LUCIFER/Webscout/issues">Report Bug</a> •
+    <a href="https://github.com/OE-LUCIFER/Webscout/issues">Request Feature</a>
+  </p>
+</div>