python-proxy-headers 0.1.0__tar.gz → 0.2.0__tar.gz

python_proxy_headers-0.2.0/PKG-INFO

@@ -0,0 +1,176 @@
+Metadata-Version: 2.4
+Name: python-proxy-headers
+Version: 0.2.0
+Summary: Handle custom proxy headers for http & https requests in various python libraries
+Author-email: ProxyMesh <support@proxymesh.com>
+Project-URL: Homepage, https://github.com/proxymesh/python-proxy-headers
+Project-URL: Changelog, https://github.com/proxymesh/python-proxy-headers/commits/main/
+Project-URL: Issues, https://github.com/proxymesh/python-proxy-headers/issues
+Project-URL: Documentation, https://python-proxy-headers.readthedocs.io/en/latest/
+Project-URL: ProxyMesh, https://proxymesh.com
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Classifier: License :: OSI Approved :: BSD License
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# Python Proxy Headers
+
+[![Documentation Status](https://readthedocs.org/projects/python-proxy-headers/badge/?version=latest)](https://python-proxy-headers.readthedocs.io/en/latest/?badge=latest)
+[![PyPI version](https://badge.fury.io/py/python-proxy-headers.svg)](https://badge.fury.io/py/python-proxy-headers)
+
+Extensions for Python HTTP libraries to support **sending and receiving custom proxy headers** during HTTPS CONNECT tunneling.
+
+## The Problem
+
+When making HTTPS requests through a proxy, the connection is established via a CONNECT tunnel. During this process:
+
+1. **Sending headers to the proxy** - Most Python HTTP libraries don't provide an easy way to send custom headers (like `X-ProxyMesh-Country`) to the proxy server during the CONNECT handshake.
+
+2. **Receiving headers from the proxy** - The proxy's response headers from the CONNECT request are typically discarded, making it impossible to read custom headers (like `X-ProxyMesh-IP`) that the proxy sends back.
+
+This library solves both problems for popular Python HTTP libraries.
+
+## Supported Libraries
+
+| Library | Module | Use Case |
+|---------|--------|----------|
+| [urllib3](https://python-proxy-headers.readthedocs.io/en/latest/urllib3.html) | `urllib3_proxy_manager` | Low-level HTTP client |
+| [requests](https://python-proxy-headers.readthedocs.io/en/latest/requests.html) | `requests_adapter` | Simple HTTP requests |
+| [aiohttp](https://python-proxy-headers.readthedocs.io/en/latest/aiohttp.html) | `aiohttp_proxy` | Async HTTP client |
+| [httpx](https://python-proxy-headers.readthedocs.io/en/latest/httpx.html) | `httpx_proxy` | Modern HTTP client |
+| [pycurl](https://python-proxy-headers.readthedocs.io/en/latest/pycurl.html) | `pycurl_proxy` | libcurl bindings |
+| [cloudscraper](https://python-proxy-headers.readthedocs.io/en/latest/cloudscraper.html) | `cloudscraper_proxy` | Cloudflare bypass |
+| [autoscraper](https://python-proxy-headers.readthedocs.io/en/latest/autoscraper.html) | `autoscraper_proxy` | Automatic web scraping |
+
+## Installation
+
+```bash
+pip install python-proxy-headers
+```
+
+Then install the HTTP library you want to use (e.g., `pip install requests`).
+
+> **Note:** This package has no dependencies by default - install only what you need.
+
+## Quick Start
+
+### requests
+
+```python
+from python_proxy_headers.requests_adapter import ProxySession
+
+with ProxySession(proxy_headers={'X-ProxyMesh-Country': 'US'}) as session:
+    session.proxies = {'https': 'http://user:pass@proxy.example.com:8080'}
+    response = session.get('https://httpbin.org/ip')
+    
+    # Proxy headers are merged into response.headers
+    print(response.headers.get('X-ProxyMesh-IP'))
+```
+
+### httpx
+
+```python
+from python_proxy_headers.httpx_proxy import get
+
+response = get(
+    'https://httpbin.org/ip',
+    proxy='http://user:pass@proxy.example.com:8080'
+)
+
+# Proxy CONNECT response headers are merged into response.headers
+print(response.headers.get('X-ProxyMesh-IP'))
+```
+
+### aiohttp
+
+```python
+import asyncio
+from python_proxy_headers.aiohttp_proxy import ProxyClientSession
+
+async def main():
+    async with ProxyClientSession() as session:
+        async with session.get(
+            'https://httpbin.org/ip',
+            proxy='http://user:pass@proxy.example.com:8080'
+        ) as response:
+            # Proxy headers merged into response.headers
+            print(response.headers.get('X-ProxyMesh-IP'))
+
+asyncio.run(main())
+```
+
+### pycurl (low-level)
+
+```python
+import pycurl
+from python_proxy_headers.pycurl_proxy import set_proxy_headers, HeaderCapture
+
+c = pycurl.Curl()
+c.setopt(pycurl.URL, 'https://httpbin.org/ip')
+c.setopt(pycurl.PROXY, 'http://proxy.example.com:8080')
+
+# Add these two lines to any existing pycurl code
+set_proxy_headers(c, {'X-ProxyMesh-Country': 'US'})
+capture = HeaderCapture(c)
+
+c.perform()
+
+print(capture.proxy_headers)  # Headers from proxy CONNECT response
+c.close()
+```
+
+### cloudscraper
+
+```python
+from python_proxy_headers.cloudscraper_proxy import create_scraper
+
+# Drop-in replacement for cloudscraper.create_scraper()
+scraper = create_scraper(proxy_headers={'X-ProxyMesh-Country': 'US'})
+scraper.proxies = {'https': 'http://proxy.example.com:8080'}
+
+response = scraper.get('https://example.com')
+# All CloudScraper features (Cloudflare bypass) preserved
+```
+
+## Testing
+
+A test harness is included to verify proxy header functionality:
+
+```bash
+# Set your proxy
+export PROXY_URL='http://user:pass@proxy.example.com:8080'
+
+# Test all modules
+python test_proxy_headers.py
+
+# Test specific modules
+python test_proxy_headers.py requests httpx
+
+# Verbose output (show header values)
+python test_proxy_headers.py -v
+```
+
+## Documentation
+
+For detailed documentation, API reference, and more examples:
+
+- **Full Documentation:** [python-proxy-headers.readthedocs.io](https://python-proxy-headers.readthedocs.io/en/latest/)
+- **Example Code:** [proxy-examples for Python](https://github.com/proxymesh/proxy-examples/tree/main/python)
+
+## Related Projects
+
+- **[scrapy-proxy-headers](https://github.com/proxymesh/scrapy-proxy-headers)** - Proxy header support for Scrapy
+
+## About
+
+Created by [ProxyMesh](https://proxymesh.com) to help our customers use custom headers to control proxy behavior. Works with any proxy that supports custom headers.
+
+## License
+
+MIT License

python_proxy_headers-0.2.0/README.md

@@ -0,0 +1,155 @@
+# Python Proxy Headers
+
+[![Documentation Status](https://readthedocs.org/projects/python-proxy-headers/badge/?version=latest)](https://python-proxy-headers.readthedocs.io/en/latest/?badge=latest)
+[![PyPI version](https://badge.fury.io/py/python-proxy-headers.svg)](https://badge.fury.io/py/python-proxy-headers)
+
+Extensions for Python HTTP libraries to support **sending and receiving custom proxy headers** during HTTPS CONNECT tunneling.
+
+## The Problem
+
+When making HTTPS requests through a proxy, the connection is established via a CONNECT tunnel. During this process:
+
+1. **Sending headers to the proxy** - Most Python HTTP libraries don't provide an easy way to send custom headers (like `X-ProxyMesh-Country`) to the proxy server during the CONNECT handshake.
+
+2. **Receiving headers from the proxy** - The proxy's response headers from the CONNECT request are typically discarded, making it impossible to read custom headers (like `X-ProxyMesh-IP`) that the proxy sends back.
+
+This library solves both problems for popular Python HTTP libraries.
+
+## Supported Libraries
+
+| Library | Module | Use Case |
+|---------|--------|----------|
+| [urllib3](https://python-proxy-headers.readthedocs.io/en/latest/urllib3.html) | `urllib3_proxy_manager` | Low-level HTTP client |
+| [requests](https://python-proxy-headers.readthedocs.io/en/latest/requests.html) | `requests_adapter` | Simple HTTP requests |
+| [aiohttp](https://python-proxy-headers.readthedocs.io/en/latest/aiohttp.html) | `aiohttp_proxy` | Async HTTP client |
+| [httpx](https://python-proxy-headers.readthedocs.io/en/latest/httpx.html) | `httpx_proxy` | Modern HTTP client |
+| [pycurl](https://python-proxy-headers.readthedocs.io/en/latest/pycurl.html) | `pycurl_proxy` | libcurl bindings |
+| [cloudscraper](https://python-proxy-headers.readthedocs.io/en/latest/cloudscraper.html) | `cloudscraper_proxy` | Cloudflare bypass |
+| [autoscraper](https://python-proxy-headers.readthedocs.io/en/latest/autoscraper.html) | `autoscraper_proxy` | Automatic web scraping |
+
+## Installation
+
+```bash
+pip install python-proxy-headers
+```
+
+Then install the HTTP library you want to use (e.g., `pip install requests`).
+
+> **Note:** This package has no dependencies by default - install only what you need.
+
+## Quick Start
+
+### requests
+
+```python
+from python_proxy_headers.requests_adapter import ProxySession
+
+with ProxySession(proxy_headers={'X-ProxyMesh-Country': 'US'}) as session:
+    session.proxies = {'https': 'http://user:pass@proxy.example.com:8080'}
+    response = session.get('https://httpbin.org/ip')
+    
+    # Proxy headers are merged into response.headers
+    print(response.headers.get('X-ProxyMesh-IP'))
+```
+
+### httpx
+
+```python
+from python_proxy_headers.httpx_proxy import get
+
+response = get(
+    'https://httpbin.org/ip',
+    proxy='http://user:pass@proxy.example.com:8080'
+)
+
+# Proxy CONNECT response headers are merged into response.headers
+print(response.headers.get('X-ProxyMesh-IP'))
+```
+
+### aiohttp
+
+```python
+import asyncio
+from python_proxy_headers.aiohttp_proxy import ProxyClientSession
+
+async def main():
+    async with ProxyClientSession() as session:
+        async with session.get(
+            'https://httpbin.org/ip',
+            proxy='http://user:pass@proxy.example.com:8080'
+        ) as response:
+            # Proxy headers merged into response.headers
+            print(response.headers.get('X-ProxyMesh-IP'))
+
+asyncio.run(main())
+```
+
+### pycurl (low-level)
+
+```python
+import pycurl
+from python_proxy_headers.pycurl_proxy import set_proxy_headers, HeaderCapture
+
+c = pycurl.Curl()
+c.setopt(pycurl.URL, 'https://httpbin.org/ip')
+c.setopt(pycurl.PROXY, 'http://proxy.example.com:8080')
+
+# Add these two lines to any existing pycurl code
+set_proxy_headers(c, {'X-ProxyMesh-Country': 'US'})
+capture = HeaderCapture(c)
+
+c.perform()
+
+print(capture.proxy_headers)  # Headers from proxy CONNECT response
+c.close()
+```
+
+### cloudscraper
+
+```python
+from python_proxy_headers.cloudscraper_proxy import create_scraper
+
+# Drop-in replacement for cloudscraper.create_scraper()
+scraper = create_scraper(proxy_headers={'X-ProxyMesh-Country': 'US'})
+scraper.proxies = {'https': 'http://proxy.example.com:8080'}
+
+response = scraper.get('https://example.com')
+# All CloudScraper features (Cloudflare bypass) preserved
+```
+
+## Testing
+
+A test harness is included to verify proxy header functionality:
+
+```bash
+# Set your proxy
+export PROXY_URL='http://user:pass@proxy.example.com:8080'
+
+# Test all modules
+python test_proxy_headers.py
+
+# Test specific modules
+python test_proxy_headers.py requests httpx
+
+# Verbose output (show header values)
+python test_proxy_headers.py -v
+```
+
+## Documentation
+
+For detailed documentation, API reference, and more examples:
+
+- **Full Documentation:** [python-proxy-headers.readthedocs.io](https://python-proxy-headers.readthedocs.io/en/latest/)
+- **Example Code:** [proxy-examples for Python](https://github.com/proxymesh/proxy-examples/tree/main/python)
+
+## Related Projects
+
+- **[scrapy-proxy-headers](https://github.com/proxymesh/scrapy-proxy-headers)** - Proxy header support for Scrapy
+
+## About
+
+Created by [ProxyMesh](https://proxymesh.com) to help our customers use custom headers to control proxy behavior. Works with any proxy that supports custom headers.
+
+## License
+
+MIT License

{python_proxy_headers-0.1.0 → python_proxy_headers-0.2.0}/pyproject.toml

@@ -4,11 +4,11 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "python-proxy-headers"
-version = "0.1.0"
+version = "0.2.0"
 authors = [
   { name="ProxyMesh", email="support@proxymesh.com" },
 ]
-description = "Handle custom proxy headers for http requests in various python libraries"
+description = "Handle custom proxy headers for http & https requests in various python libraries"
 readme = "README.md"
 requires-python = ">=3.8"
 classifiers = [
@@ -24,3 +24,5 @@ classifiers = [
 Homepage = "https://github.com/proxymesh/python-proxy-headers"
 Changelog = "https://github.com/proxymesh/python-proxy-headers/commits/main/"
 Issues = "https://github.com/proxymesh/python-proxy-headers/issues"
+Documentation = "https://python-proxy-headers.readthedocs.io/en/latest/"
+ProxyMesh = "https://proxymesh.com"

python_proxy_headers-0.2.0/python_proxy_headers/autoscraper_proxy.py

@@ -0,0 +1,344 @@
+"""
+AutoScraper extension for sending and receiving proxy headers.
+
+This module provides an AutoScraper subclass that enables:
+1. Sending custom headers to proxy servers during CONNECT
+2. Using our ProxySession for all HTTP requests
+
+Example usage:
+    from python_proxy_headers.autoscraper_proxy import ProxyAutoScraper
+
+    scraper = ProxyAutoScraper(proxy_headers={'X-ProxyMesh-Country': 'US'})
+    
+    # Build with proxy
+    result = scraper.build(
+        url='https://example.com',
+        wanted_list=['Example Domain'],
+        request_args={'proxies': {'https': 'http://proxy:8080'}}
+    )
+    
+    # Get results with proxy
+    result = scraper.get_result_similar(
+        url='https://other-example.com',
+        request_args={'proxies': {'https': 'http://proxy:8080'}}
+    )
+"""
+
+from typing import Dict, List, Optional, Any
+from urllib.parse import urlparse
+
+try:
+    from autoscraper import AutoScraper
+except ImportError:
+    raise ImportError(
+        "autoscraper is required for this module. "
+        "Install it with: pip install autoscraper"
+    )
+
+from .requests_adapter import ProxySession
+
+
+class ProxyAutoScraper(AutoScraper):
+    """
+    AutoScraper with proxy header support.
+    
+    This class extends AutoScraper to use our ProxySession for HTTP requests,
+    enabling custom proxy headers to be sent during CONNECT tunneling.
+    
+    Args:
+        proxy_headers: Dict of headers to send to proxy servers
+        stack_list: Initial stack list (rules) for the scraper
+    
+    Example:
+        scraper = ProxyAutoScraper(proxy_headers={'X-ProxyMesh-Country': 'US'})
+        
+        result = scraper.build(
+            url='https://finance.yahoo.com/quote/AAPL/',
+            wanted_list=['Apple Inc.'],
+            request_args={'proxies': {'https': 'http://proxy:8080'}}
+        )
+        
+        # Use the learned rules on another page
+        result = scraper.get_result_similar(
+            url='https://finance.yahoo.com/quote/GOOG/',
+            request_args={'proxies': {'https': 'http://proxy:8080'}}
+        )
+    """
+    
+    def __init__(
+        self,
+        proxy_headers: Optional[Dict[str, str]] = None,
+        stack_list: Optional[List] = None
+    ):
+        super().__init__(stack_list=stack_list)
+        self._proxy_headers = proxy_headers or {}
+        self._session: Optional[ProxySession] = None
+    
+    def _get_session(self) -> ProxySession:
+        """Get or create the ProxySession."""
+        if self._session is None:
+            self._session = ProxySession(proxy_headers=self._proxy_headers)
+        return self._session
+    
+    def set_proxy_headers(self, proxy_headers: Dict[str, str]):
+        """
+        Update the proxy headers.
+        
+        This will close the current session and create a new one with
+        the updated headers on the next request.
+        
+        Args:
+            proxy_headers: New proxy headers to use
+        """
+        self._proxy_headers = proxy_headers
+        if self._session is not None:
+            self._session.close()
+            self._session = None
+    
+    def close(self):
+        """Close the underlying session."""
+        if self._session is not None:
+            self._session.close()
+            self._session = None
+    
+    def __enter__(self):
+        return self
+    
+    def __exit__(self, *args):
+        self.close()
+    
+    @classmethod
+    def _fetch_html(cls, url, request_args=None):
+        """
+        Fetch HTML from URL using the standard requests.
+        
+        Note: This is the class method from parent. For proxy header support,
+        use instance methods which use the ProxySession.
+        """
+        # Fall back to parent implementation for class method calls
+        return super()._fetch_html(url, request_args)
+    
+    def _fetch_html_with_proxy(self, url: str, request_args: Optional[Dict] = None) -> str:
+        """
+        Fetch HTML from URL using ProxySession with proxy header support.
+        
+        Args:
+            url: URL to fetch
+            request_args: Additional request arguments (proxies, headers, etc.)
+            
+        Returns:
+            HTML content as string
+        """
+        request_args = request_args or {}
+        
+        # Build headers
+        headers = dict(self.request_headers)
+        if url:
+            headers["Host"] = urlparse(url).netloc
+        
+        user_headers = request_args.pop("headers", {})
+        headers.update(user_headers)
+        
+        # Use our ProxySession
+        session = self._get_session()
+        
+        # Copy session-level settings if not in request_args
+        if 'proxies' in request_args:
+            session.proxies.update(request_args.pop('proxies'))
+        
+        res = session.get(url, headers=headers, **request_args)
+        
+        # Handle encoding
+        if res.encoding == "ISO-8859-1" and "ISO-8859-1" not in res.headers.get(
+            "Content-Type", ""
+        ):
+            res.encoding = res.apparent_encoding
+        
+        return res.text
+    
+    def _get_soup_with_proxy(self, url=None, html=None, request_args=None):
+        """
+        Get BeautifulSoup object using ProxySession.
+        
+        Args:
+            url: URL to fetch (optional if html is provided)
+            html: HTML string (optional if url is provided)
+            request_args: Additional request arguments
+            
+        Returns:
+            BeautifulSoup object
+        """
+        from html import unescape
+        from bs4 import BeautifulSoup
+        from autoscraper.utils import normalize
+        
+        if html:
+            html = normalize(unescape(html))
+            return BeautifulSoup(html, "lxml")
+        
+        html = self._fetch_html_with_proxy(url, request_args)
+        html = normalize(unescape(html))
+        
+        return BeautifulSoup(html, "lxml")
+    
+    def build(
+        self,
+        url: Optional[str] = None,
+        wanted_list: Optional[List] = None,
+        wanted_dict: Optional[Dict] = None,
+        html: Optional[str] = None,
+        request_args: Optional[Dict] = None,
+        update: bool = False,
+        text_fuzz_ratio: float = 1.0,
+    ) -> List:
+        """
+        Build scraping rules with proxy header support.
+        
+        Same as AutoScraper.build() but uses ProxySession for requests.
+        
+        Parameters:
+            url: URL of the target web page
+            wanted_list: List of needed contents to be scraped
+            wanted_dict: Dict of needed contents (keys are aliases)
+            html: HTML string (alternative to URL)
+            request_args: Request arguments including proxies
+            update: If True, add to existing rules
+            text_fuzz_ratio: Fuzziness ratio for matching
+            
+        Returns:
+            List of similar results
+        """
+        from html import unescape
+        from autoscraper.utils import normalize, unique_hashable, unique_stack_list
+        
+        if not wanted_list and not (wanted_dict and any(wanted_dict.values())):
+            raise ValueError("No targets were supplied")
+        
+        # Use our proxy-aware soup getter
+        soup = self._get_soup_with_proxy(url=url, html=html, request_args=request_args)
+        
+        result_list = []
+        
+        if update is False:
+            self.stack_list = []
+        
+        if wanted_list:
+            wanted_dict = {"": wanted_list}
+        
+        wanted_list = []
+        
+        for alias, wanted_items in wanted_dict.items():
+            wanted_items = [normalize(w) for w in wanted_items]
+            wanted_list += wanted_items
+            
+            for wanted in wanted_items:
+                children = self._get_children(soup, wanted, url, text_fuzz_ratio)
+                
+                for child in children:
+                    result, stack = self._get_result_for_child(child, soup, url)
+                    stack["alias"] = alias
+                    result_list += result
+                    self.stack_list.append(stack)
+        
+        result_list = [item.text for item in result_list]
+        result_list = unique_hashable(result_list)
+        
+        self.stack_list = unique_stack_list(self.stack_list)
+        return result_list
+    
+    def get_result_similar(
+        self,
+        url: Optional[str] = None,
+        html: Optional[str] = None,
+        soup=None,
+        request_args: Optional[Dict] = None,
+        grouped: bool = False,
+        group_by_alias: bool = False,
+        unique: Optional[bool] = None,
+        attr_fuzz_ratio: float = 1.0,
+        keep_blank: bool = False,
+        keep_order: bool = False,
+        contain_sibling_leaves: bool = False,
+    ):
+        """
+        Get similar results with proxy header support.
+        
+        Same as AutoScraper.get_result_similar() but uses ProxySession.
+        """
+        if soup is None and url is not None:
+            soup = self._get_soup_with_proxy(url=url, html=html, request_args=request_args)
+        
+        return super().get_result_similar(
+            url=url,
+            html=html,
+            soup=soup,
+            request_args=None,  # Already fetched
+            grouped=grouped,
+            group_by_alias=group_by_alias,
+            unique=unique,
+            attr_fuzz_ratio=attr_fuzz_ratio,
+            keep_blank=keep_blank,
+            keep_order=keep_order,
+            contain_sibling_leaves=contain_sibling_leaves,
+        )
+    
+    def get_result_exact(
+        self,
+        url: Optional[str] = None,
+        html: Optional[str] = None,
+        soup=None,
+        request_args: Optional[Dict] = None,
+        grouped: bool = False,
+        group_by_alias: bool = False,
+        unique: Optional[bool] = None,
+        attr_fuzz_ratio: float = 1.0,
+        keep_blank: bool = False,
+    ):
+        """
+        Get exact results with proxy header support.
+        
+        Same as AutoScraper.get_result_exact() but uses ProxySession.
+        """
+        if soup is None and url is not None:
+            soup = self._get_soup_with_proxy(url=url, html=html, request_args=request_args)
+        
+        return super().get_result_exact(
+            url=url,
+            html=html,
+            soup=soup,
+            request_args=None,  # Already fetched
+            grouped=grouped,
+            group_by_alias=group_by_alias,
+            unique=unique,
+            attr_fuzz_ratio=attr_fuzz_ratio,
+            keep_blank=keep_blank,
+        )
+    
+    def get_result(
+        self,
+        url: Optional[str] = None,
+        html: Optional[str] = None,
+        request_args: Optional[Dict] = None,
+        grouped: bool = False,
+        group_by_alias: bool = False,
+        unique: Optional[bool] = None,
+        attr_fuzz_ratio: float = 1.0,
+    ):
+        """
+        Get similar and exact results with proxy header support.
+        
+        Same as AutoScraper.get_result() but uses ProxySession.
+        """
+        soup = self._get_soup_with_proxy(url=url, html=html, request_args=request_args)
+        
+        args = dict(
+            url=url,
+            soup=soup,
+            grouped=grouped,
+            group_by_alias=group_by_alias,
+            unique=unique,
+            attr_fuzz_ratio=attr_fuzz_ratio,
+        )
+        similar = self.get_result_similar(**args)
+        exact = self.get_result_exact(**args)
+        return similar, exact