scrapy-proxy-headers 0.1.2__tar.gz → 0.1.4__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {scrapy_proxy_headers-0.1.2 → scrapy-proxy-headers-0.1.4}/PKG-INFO +15 -8
- {scrapy_proxy_headers-0.1.2 → scrapy-proxy-headers-0.1.4}/README.md +12 -6
- {scrapy_proxy_headers-0.1.2 → scrapy-proxy-headers-0.1.4}/pyproject.toml +2 -1
- {scrapy_proxy_headers-0.1.2 → scrapy-proxy-headers-0.1.4}/scrapy_proxy_headers/agent.py +36 -17
- scrapy-proxy-headers-0.1.4/scrapy_proxy_headers/download_handler.py +37 -0
- {scrapy_proxy_headers-0.1.2 → scrapy-proxy-headers-0.1.4}/scrapy_proxy_headers.egg-info/PKG-INFO +15 -8
- scrapy_proxy_headers-0.1.2/scrapy_proxy_headers/download_handler.py +0 -15
- {scrapy_proxy_headers-0.1.2 → scrapy-proxy-headers-0.1.4}/LICENSE +0 -0
- {scrapy_proxy_headers-0.1.2 → scrapy-proxy-headers-0.1.4}/scrapy_proxy_headers/__init__.py +0 -0
- {scrapy_proxy_headers-0.1.2 → scrapy-proxy-headers-0.1.4}/scrapy_proxy_headers.egg-info/SOURCES.txt +0 -0
- {scrapy_proxy_headers-0.1.2 → scrapy-proxy-headers-0.1.4}/scrapy_proxy_headers.egg-info/dependency_links.txt +0 -0
- {scrapy_proxy_headers-0.1.2 → scrapy-proxy-headers-0.1.4}/scrapy_proxy_headers.egg-info/top_level.txt +0 -0
- {scrapy_proxy_headers-0.1.2 → scrapy-proxy-headers-0.1.4}/setup.cfg +0 -0
|
@@ -1,10 +1,11 @@
|
|
|
1
|
-
Metadata-Version: 2.
|
|
1
|
+
Metadata-Version: 2.1
|
|
2
2
|
Name: scrapy-proxy-headers
|
|
3
|
-
Version: 0.1.
|
|
3
|
+
Version: 0.1.4
|
|
4
4
|
Summary: Add custom proxy headers to HTTPS requests in Scrapy
|
|
5
5
|
Author-email: ProxyMesh <support@proxymesh.com>
|
|
6
6
|
Project-URL: Homepage, https://github.com/proxymesh/scrapy-proxy-headers
|
|
7
7
|
Project-URL: Changelog, https://github.com/proxymesh/scrapy-proxy-headers/commits/main/
|
|
8
|
+
Project-URL: Issues, https://github.com/proxymesh/scrapy-proxy-headers/issues
|
|
8
9
|
Classifier: Programming Language :: Python :: 3
|
|
9
10
|
Classifier: Operating System :: OS Independent
|
|
10
11
|
Classifier: License :: OSI Approved :: BSD License
|
|
@@ -15,16 +16,20 @@ Requires-Python: >=3.8
|
|
|
15
16
|
Description-Content-Type: text/markdown
|
|
16
17
|
License-File: LICENSE
|
|
17
18
|
|
|
18
|
-
|
|
19
|
+
# Scrapy Proxy Headers
|
|
20
|
+
|
|
21
|
+
The `scrapy-proxy-headers` package is designed for adding proxy headers to HTTPS requests in [Scrapy](https://scrapy.org/).
|
|
19
22
|
|
|
20
23
|
In normal usage, custom headers put in `request.headers` cannot be read by a proxy when you make a HTTPS request, because the headers are encrypted and passed through the proxy tunnel, along with the rest of the request body. You can read more about this at [Proxy Server Requests over HTTPS](https://docs.proxymesh.com/article/145-proxy-server-requests-over-https).
|
|
21
24
|
|
|
22
|
-
Because Scrapy does not have a good way to pass custom headers to a proxy when you make HTTPS requests, we at [ProxyMesh](https://proxymesh.com) made this extension to support our customers that use Scrapy and want to use custom headers to control our proxy behavior. But this extension can work for
|
|
25
|
+
Because Scrapy does not have a good way to pass custom headers to a proxy when you make HTTPS requests, we at [ProxyMesh](https://proxymesh.com) made this extension to support our customers that use Scrapy and want to use custom headers to control our proxy behavior. But this extension can work for handling custom headers through any proxy.
|
|
26
|
+
|
|
27
|
+
## Installation
|
|
23
28
|
|
|
24
29
|
To use this extension, do the following:
|
|
25
30
|
|
|
26
|
-
1. `pip install
|
|
27
|
-
2. In your Scrapy `settings.py`, add the following
|
|
31
|
+
1. `pip install scrapy-proxy-headers`
|
|
32
|
+
2. In your Scrapy `settings.py`, add the following:
|
|
28
33
|
|
|
29
34
|
```python
|
|
30
35
|
DOWNLOAD_HANDLERS = {
|
|
@@ -32,10 +37,12 @@ DOWNLOAD_HANDLERS = {
|
|
|
32
37
|
}
|
|
33
38
|
```
|
|
34
39
|
|
|
35
|
-
|
|
40
|
+
## Usage
|
|
41
|
+
|
|
42
|
+
When you want make a request with a custom proxy header, instead of using `request.headers`, use `request.meta["proxy_headers"]` like this:
|
|
36
43
|
|
|
37
44
|
```python
|
|
38
45
|
request.meta["proxy_headers"] = {"X-ProxyMesh-Country": "US"}
|
|
39
46
|
```
|
|
40
47
|
|
|
41
|
-
|
|
48
|
+
Any response headers that might come from the proxy will be saved in `response.headers`, as in `response.headers["X-ProxyMesh-IP"]`.
|
|
@@ -1,13 +1,17 @@
|
|
|
1
|
-
|
|
1
|
+
# Scrapy Proxy Headers
|
|
2
|
+
|
|
3
|
+
The `scrapy-proxy-headers` package is designed for adding proxy headers to HTTPS requests in [Scrapy](https://scrapy.org/).
|
|
2
4
|
|
|
3
5
|
In normal usage, custom headers put in `request.headers` cannot be read by a proxy when you make a HTTPS request, because the headers are encrypted and passed through the proxy tunnel, along with the rest of the request body. You can read more about this at [Proxy Server Requests over HTTPS](https://docs.proxymesh.com/article/145-proxy-server-requests-over-https).
|
|
4
6
|
|
|
5
|
-
Because Scrapy does not have a good way to pass custom headers to a proxy when you make HTTPS requests, we at [ProxyMesh](https://proxymesh.com) made this extension to support our customers that use Scrapy and want to use custom headers to control our proxy behavior. But this extension can work for
|
|
7
|
+
Because Scrapy does not have a good way to pass custom headers to a proxy when you make HTTPS requests, we at [ProxyMesh](https://proxymesh.com) made this extension to support our customers that use Scrapy and want to use custom headers to control our proxy behavior. But this extension can work for handling custom headers through any proxy.
|
|
8
|
+
|
|
9
|
+
## Installation
|
|
6
10
|
|
|
7
11
|
To use this extension, do the following:
|
|
8
12
|
|
|
9
|
-
1. `pip install
|
|
10
|
-
2. In your Scrapy `settings.py`, add the following
|
|
13
|
+
1. `pip install scrapy-proxy-headers`
|
|
14
|
+
2. In your Scrapy `settings.py`, add the following:
|
|
11
15
|
|
|
12
16
|
```python
|
|
13
17
|
DOWNLOAD_HANDLERS = {
|
|
@@ -15,10 +19,12 @@ DOWNLOAD_HANDLERS = {
|
|
|
15
19
|
}
|
|
16
20
|
```
|
|
17
21
|
|
|
18
|
-
|
|
22
|
+
## Usage
|
|
23
|
+
|
|
24
|
+
When you want make a request with a custom proxy header, instead of using `request.headers`, use `request.meta["proxy_headers"]` like this:
|
|
19
25
|
|
|
20
26
|
```python
|
|
21
27
|
request.meta["proxy_headers"] = {"X-ProxyMesh-Country": "US"}
|
|
22
28
|
```
|
|
23
29
|
|
|
24
|
-
|
|
30
|
+
Any response headers that might come from the proxy will be saved in `response.headers`, as in `response.headers["X-ProxyMesh-IP"]`.
|
|
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|
|
4
4
|
|
|
5
5
|
[project]
|
|
6
6
|
name = "scrapy-proxy-headers"
|
|
7
|
-
version = "0.1.
|
|
7
|
+
version = "0.1.4"
|
|
8
8
|
authors = [
|
|
9
9
|
{ name="ProxyMesh", email="support@proxymesh.com" },
|
|
10
10
|
]
|
|
@@ -23,3 +23,4 @@ classifiers = [
|
|
|
23
23
|
[project.urls]
|
|
24
24
|
Homepage = "https://github.com/proxymesh/scrapy-proxy-headers"
|
|
25
25
|
Changelog = "https://github.com/proxymesh/scrapy-proxy-headers/commits/main/"
|
|
26
|
+
Issues = "https://github.com/proxymesh/scrapy-proxy-headers/issues"
|
|
@@ -1,6 +1,8 @@
|
|
|
1
|
-
|
|
1
|
+
import re
|
|
2
|
+
from scrapy.core.downloader.handlers.http11 import TunnelingAgent, TunnelingTCP4ClientEndpoint, ScrapyAgent
|
|
2
3
|
from scrapy.core.downloader.webclient import _parse
|
|
3
4
|
from scrapy.utils.python import to_bytes
|
|
5
|
+
from scrapy.http import Headers, Response
|
|
4
6
|
|
|
5
7
|
def tunnel_request_data_with_headers(host: str, port: int, **proxy_headers) -> bytes:
|
|
6
8
|
r"""
|
|
@@ -54,6 +56,19 @@ class TunnelingHeadersTCP4ClientEndpoint(TunnelingTCP4ClientEndpoint):
|
|
|
54
56
|
protocol.dataReceived = self.processProxyResponse # type: ignore[method-assign]
|
|
55
57
|
self._protocol = protocol
|
|
56
58
|
return protocol
|
|
59
|
+
|
|
60
|
+
def processProxyResponse(self, data: bytes):
|
|
61
|
+
# data might have proxy headers, looks like
|
|
62
|
+
# b'HTTP/1.1 200 Connection established\r\nProxy-Header: VALUE\r\n\r\n'
|
|
63
|
+
response_headers = {}
|
|
64
|
+
|
|
65
|
+
for line in data.split(b'\r\n'):
|
|
66
|
+
if b':' in line:
|
|
67
|
+
key, val = line.split(b':', 1)
|
|
68
|
+
response_headers[key.strip()] = val.strip()
|
|
69
|
+
# save for endpoing & agent
|
|
70
|
+
self._proxy_response_headers = Headers(response_headers)
|
|
71
|
+
return super(TunnelingHeadersTCP4ClientEndpoint, self).processProxyResponse(data)
|
|
57
72
|
|
|
58
73
|
class TunnelingHeadersAgent(TunnelingAgent):
|
|
59
74
|
"""An agent that uses a L{TunnelingTCP4ClientEndpoint} to make HTTPS
|
|
@@ -70,7 +85,8 @@ class TunnelingHeadersAgent(TunnelingAgent):
|
|
|
70
85
|
self._proxy_headers = proxy_headers
|
|
71
86
|
|
|
72
87
|
def _getEndpoint(self, uri):
|
|
73
|
-
|
|
88
|
+
# save endpoint for agent to get proxy_response_headers
|
|
89
|
+
self._endpoint = TunnelingHeadersTCP4ClientEndpoint(
|
|
74
90
|
reactor=self._reactor,
|
|
75
91
|
host=uri.host,
|
|
76
92
|
port=uri.port,
|
|
@@ -80,31 +96,34 @@ class TunnelingHeadersAgent(TunnelingAgent):
|
|
|
80
96
|
bindAddress=self._endpointFactory._bindAddress,
|
|
81
97
|
**self._proxy_headers
|
|
82
98
|
)
|
|
99
|
+
return self._endpoint
|
|
83
100
|
|
|
84
101
|
class ScrapyProxyHeadersAgent(ScrapyAgent):
|
|
85
102
|
_TunnelingAgent = TunnelingHeadersAgent
|
|
103
|
+
|
|
104
|
+
def __init__(self, *args, **kwargs):
|
|
105
|
+
super().__init__(*args, **kwargs)
|
|
106
|
+
self._agent = None
|
|
86
107
|
|
|
87
108
|
def _get_agent(self, request, timeout: float):
|
|
88
|
-
|
|
109
|
+
self._agent = super()._get_agent(request, timeout)
|
|
89
110
|
|
|
90
111
|
proxy = request.meta.get("proxy")
|
|
91
112
|
proxy_headers = request.meta.get('proxy_headers')
|
|
92
113
|
if proxy and proxy_headers:
|
|
93
114
|
scheme = _parse(request.url)[0]
|
|
94
115
|
if scheme == b"https":
|
|
95
|
-
|
|
116
|
+
self._agent.set_proxy_headers(proxy_headers)
|
|
96
117
|
|
|
97
|
-
return
|
|
118
|
+
return self._agent
|
|
98
119
|
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
)
|
|
110
|
-
return agent.download_request(request)
|
|
120
|
+
def _cb_bodydone(self, result, request, url: str):
|
|
121
|
+
r = super()._cb_bodydone(result, request, url)
|
|
122
|
+
if isinstance(r, Response):
|
|
123
|
+
if self._agent and hasattr(self._agent, '_endpoint'):
|
|
124
|
+
proxy_response_headers = getattr(self._agent._endpoint, '_proxy_response_headers', None)
|
|
125
|
+
if proxy_response_headers:
|
|
126
|
+
r.headers.update(proxy_response_headers)
|
|
127
|
+
# save this for download handler
|
|
128
|
+
r._proxy_response_headers = proxy_response_headers
|
|
129
|
+
return r
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
from scrapy.core.downloader.handlers.http11 import HTTP11DownloadHandler
|
|
2
|
+
from scrapy_proxy_headers.agent import ScrapyProxyHeadersAgent
|
|
3
|
+
|
|
4
|
+
class HTTP11ProxyDownloadHandler(HTTP11DownloadHandler):
|
|
5
|
+
def __init__(self, *args, **kwargs):
|
|
6
|
+
super().__init__(*args, **kwargs)
|
|
7
|
+
self._proxy_headers_by_proxy = {}
|
|
8
|
+
|
|
9
|
+
def download_request(self, request, spider):
|
|
10
|
+
"""Return a deferred for the HTTP download"""
|
|
11
|
+
agent = ScrapyProxyHeadersAgent(
|
|
12
|
+
contextFactory=self._contextFactory,
|
|
13
|
+
pool=self._pool,
|
|
14
|
+
maxsize=getattr(spider, "download_maxsize", self._default_maxsize),
|
|
15
|
+
warnsize=getattr(spider, "download_warnsize", self._default_warnsize),
|
|
16
|
+
fail_on_dataloss=self._fail_on_dataloss,
|
|
17
|
+
crawler=self._crawler,
|
|
18
|
+
)
|
|
19
|
+
response = agent.download_request(request)
|
|
20
|
+
proxy = request.meta.get("proxy")
|
|
21
|
+
|
|
22
|
+
if proxy:
|
|
23
|
+
# we need to do all this because the proxy tunnels can get re-used
|
|
24
|
+
# when that happens, the proxy headers are not available in subsequent responses
|
|
25
|
+
# so we need to save the proxy headers by the proxy, from the first tunnel response
|
|
26
|
+
# so we can add them to subsequent responses
|
|
27
|
+
def callback(response):
|
|
28
|
+
if hasattr(response, '_proxy_response_headers'):
|
|
29
|
+
self._proxy_headers_by_proxy[proxy] = response._proxy_response_headers
|
|
30
|
+
|
|
31
|
+
if proxy in self._proxy_headers_by_proxy:
|
|
32
|
+
response.headers.update(self._proxy_headers_by_proxy[proxy])
|
|
33
|
+
|
|
34
|
+
return response
|
|
35
|
+
|
|
36
|
+
response.addCallback(callback)
|
|
37
|
+
return response
|
{scrapy_proxy_headers-0.1.2 → scrapy-proxy-headers-0.1.4}/scrapy_proxy_headers.egg-info/PKG-INFO
RENAMED
|
@@ -1,10 +1,11 @@
|
|
|
1
|
-
Metadata-Version: 2.
|
|
1
|
+
Metadata-Version: 2.1
|
|
2
2
|
Name: scrapy-proxy-headers
|
|
3
|
-
Version: 0.1.
|
|
3
|
+
Version: 0.1.4
|
|
4
4
|
Summary: Add custom proxy headers to HTTPS requests in Scrapy
|
|
5
5
|
Author-email: ProxyMesh <support@proxymesh.com>
|
|
6
6
|
Project-URL: Homepage, https://github.com/proxymesh/scrapy-proxy-headers
|
|
7
7
|
Project-URL: Changelog, https://github.com/proxymesh/scrapy-proxy-headers/commits/main/
|
|
8
|
+
Project-URL: Issues, https://github.com/proxymesh/scrapy-proxy-headers/issues
|
|
8
9
|
Classifier: Programming Language :: Python :: 3
|
|
9
10
|
Classifier: Operating System :: OS Independent
|
|
10
11
|
Classifier: License :: OSI Approved :: BSD License
|
|
@@ -15,16 +16,20 @@ Requires-Python: >=3.8
|
|
|
15
16
|
Description-Content-Type: text/markdown
|
|
16
17
|
License-File: LICENSE
|
|
17
18
|
|
|
18
|
-
|
|
19
|
+
# Scrapy Proxy Headers
|
|
20
|
+
|
|
21
|
+
The `scrapy-proxy-headers` package is designed for adding proxy headers to HTTPS requests in [Scrapy](https://scrapy.org/).
|
|
19
22
|
|
|
20
23
|
In normal usage, custom headers put in `request.headers` cannot be read by a proxy when you make a HTTPS request, because the headers are encrypted and passed through the proxy tunnel, along with the rest of the request body. You can read more about this at [Proxy Server Requests over HTTPS](https://docs.proxymesh.com/article/145-proxy-server-requests-over-https).
|
|
21
24
|
|
|
22
|
-
Because Scrapy does not have a good way to pass custom headers to a proxy when you make HTTPS requests, we at [ProxyMesh](https://proxymesh.com) made this extension to support our customers that use Scrapy and want to use custom headers to control our proxy behavior. But this extension can work for
|
|
25
|
+
Because Scrapy does not have a good way to pass custom headers to a proxy when you make HTTPS requests, we at [ProxyMesh](https://proxymesh.com) made this extension to support our customers that use Scrapy and want to use custom headers to control our proxy behavior. But this extension can work for handling custom headers through any proxy.
|
|
26
|
+
|
|
27
|
+
## Installation
|
|
23
28
|
|
|
24
29
|
To use this extension, do the following:
|
|
25
30
|
|
|
26
|
-
1. `pip install
|
|
27
|
-
2. In your Scrapy `settings.py`, add the following
|
|
31
|
+
1. `pip install scrapy-proxy-headers`
|
|
32
|
+
2. In your Scrapy `settings.py`, add the following:
|
|
28
33
|
|
|
29
34
|
```python
|
|
30
35
|
DOWNLOAD_HANDLERS = {
|
|
@@ -32,10 +37,12 @@ DOWNLOAD_HANDLERS = {
|
|
|
32
37
|
}
|
|
33
38
|
```
|
|
34
39
|
|
|
35
|
-
|
|
40
|
+
## Usage
|
|
41
|
+
|
|
42
|
+
When you want make a request with a custom proxy header, instead of using `request.headers`, use `request.meta["proxy_headers"]` like this:
|
|
36
43
|
|
|
37
44
|
```python
|
|
38
45
|
request.meta["proxy_headers"] = {"X-ProxyMesh-Country": "US"}
|
|
39
46
|
```
|
|
40
47
|
|
|
41
|
-
|
|
48
|
+
Any response headers that might come from the proxy will be saved in `response.headers`, as in `response.headers["X-ProxyMesh-IP"]`.
|
|
@@ -1,15 +0,0 @@
|
|
|
1
|
-
from scrapy.core.downloader.handlers.http11 import HTTP11DownloadHandler
|
|
2
|
-
from scrapy_proxy_headers.agent import ScrapyProxyHeadersAgent
|
|
3
|
-
|
|
4
|
-
class HTTP11ProxyDownloadHandler(HTTP11DownloadHandler):
|
|
5
|
-
def download_request(self, request, spider):
|
|
6
|
-
"""Return a deferred for the HTTP download"""
|
|
7
|
-
agent = ScrapyProxyHeadersAgent(
|
|
8
|
-
contextFactory=self._contextFactory,
|
|
9
|
-
pool=self._pool,
|
|
10
|
-
maxsize=getattr(spider, "download_maxsize", self._default_maxsize),
|
|
11
|
-
warnsize=getattr(spider, "download_warnsize", self._default_warnsize),
|
|
12
|
-
fail_on_dataloss=self._fail_on_dataloss,
|
|
13
|
-
crawler=self._crawler,
|
|
14
|
-
)
|
|
15
|
-
return agent.download_request(request)
|
|
File without changes
|
|
File without changes
|
{scrapy_proxy_headers-0.1.2 → scrapy-proxy-headers-0.1.4}/scrapy_proxy_headers.egg-info/SOURCES.txt
RENAMED
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|