scrapy-proxy-headers 0.1.2__tar.gz → 0.1.4__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,10 +1,11 @@
1
- Metadata-Version: 2.2
1
+ Metadata-Version: 2.1
2
2
  Name: scrapy-proxy-headers
3
- Version: 0.1.2
3
+ Version: 0.1.4
4
4
  Summary: Add custom proxy headers to HTTPS requests in Scrapy
5
5
  Author-email: ProxyMesh <support@proxymesh.com>
6
6
  Project-URL: Homepage, https://github.com/proxymesh/scrapy-proxy-headers
7
7
  Project-URL: Changelog, https://github.com/proxymesh/scrapy-proxy-headers/commits/main/
8
+ Project-URL: Issues, https://github.com/proxymesh/scrapy-proxy-headers/issues
8
9
  Classifier: Programming Language :: Python :: 3
9
10
  Classifier: Operating System :: OS Independent
10
11
  Classifier: License :: OSI Approved :: BSD License
@@ -15,16 +16,20 @@ Requires-Python: >=3.8
15
16
  Description-Content-Type: text/markdown
16
17
  License-File: LICENSE
17
18
 
18
- The `scrapy-proxy-headers` package is designed for adding proxy headers to HTTPS requests.
19
+ # Scrapy Proxy Headers
20
+
21
+ The `scrapy-proxy-headers` package is designed for adding proxy headers to HTTPS requests in [Scrapy](https://scrapy.org/).
19
22
 
20
23
  In normal usage, custom headers put in `request.headers` cannot be read by a proxy when you make a HTTPS request, because the headers are encrypted and passed through the proxy tunnel, along with the rest of the request body. You can read more about this at [Proxy Server Requests over HTTPS](https://docs.proxymesh.com/article/145-proxy-server-requests-over-https).
21
24
 
22
- Because Scrapy does not have a good way to pass custom headers to a proxy when you make HTTPS requests, we at [ProxyMesh](https://proxymesh.com) made this extension to support our customers that use Scrapy and want to use custom headers to control our proxy behavior. But this extension can work for any custom headers through a proxy.
25
+ Because Scrapy does not have a good way to pass custom headers to a proxy when you make HTTPS requests, we at [ProxyMesh](https://proxymesh.com) made this extension to support our customers that use Scrapy and want to use custom headers to control our proxy behavior. But this extension can work for handling custom headers through any proxy.
26
+
27
+ ## Installation
23
28
 
24
29
  To use this extension, do the following:
25
30
 
26
- 1. `pip install scrapy_proxy_headers`
27
- 2. In your Scrapy `settings.py`, add the following code:
31
+ 1. `pip install scrapy-proxy-headers`
32
+ 2. In your Scrapy `settings.py`, add the following:
28
33
 
29
34
  ```python
30
35
  DOWNLOAD_HANDLERS = {
@@ -32,10 +37,12 @@ DOWNLOAD_HANDLERS = {
32
37
  }
33
38
  ```
34
39
 
35
- 3. When you want make a request with a custom proxy header, instead of using `request.headers`, use `request.meta["proxy_headers"]` like this:
40
+ ## Usage
41
+
42
+ When you want make a request with a custom proxy header, instead of using `request.headers`, use `request.meta["proxy_headers"]` like this:
36
43
 
37
44
  ```python
38
45
  request.meta["proxy_headers"] = {"X-ProxyMesh-Country": "US"}
39
46
  ```
40
47
 
41
- Currently this package does not yet support reading custom response headers from the proxy, but that feature is coming soon.
48
+ Any response headers that might come from the proxy will be saved in `response.headers`, as in `response.headers["X-ProxyMesh-IP"]`.
@@ -1,13 +1,17 @@
1
- The `scrapy-proxy-headers` package is designed for adding proxy headers to HTTPS requests.
1
+ # Scrapy Proxy Headers
2
+
3
+ The `scrapy-proxy-headers` package is designed for adding proxy headers to HTTPS requests in [Scrapy](https://scrapy.org/).
2
4
 
3
5
  In normal usage, custom headers put in `request.headers` cannot be read by a proxy when you make a HTTPS request, because the headers are encrypted and passed through the proxy tunnel, along with the rest of the request body. You can read more about this at [Proxy Server Requests over HTTPS](https://docs.proxymesh.com/article/145-proxy-server-requests-over-https).
4
6
 
5
- Because Scrapy does not have a good way to pass custom headers to a proxy when you make HTTPS requests, we at [ProxyMesh](https://proxymesh.com) made this extension to support our customers that use Scrapy and want to use custom headers to control our proxy behavior. But this extension can work for any custom headers through a proxy.
7
+ Because Scrapy does not have a good way to pass custom headers to a proxy when you make HTTPS requests, we at [ProxyMesh](https://proxymesh.com) made this extension to support our customers that use Scrapy and want to use custom headers to control our proxy behavior. But this extension can work for handling custom headers through any proxy.
8
+
9
+ ## Installation
6
10
 
7
11
  To use this extension, do the following:
8
12
 
9
- 1. `pip install scrapy_proxy_headers`
10
- 2. In your Scrapy `settings.py`, add the following code:
13
+ 1. `pip install scrapy-proxy-headers`
14
+ 2. In your Scrapy `settings.py`, add the following:
11
15
 
12
16
  ```python
13
17
  DOWNLOAD_HANDLERS = {
@@ -15,10 +19,12 @@ DOWNLOAD_HANDLERS = {
15
19
  }
16
20
  ```
17
21
 
18
- 3. When you want make a request with a custom proxy header, instead of using `request.headers`, use `request.meta["proxy_headers"]` like this:
22
+ ## Usage
23
+
24
+ When you want make a request with a custom proxy header, instead of using `request.headers`, use `request.meta["proxy_headers"]` like this:
19
25
 
20
26
  ```python
21
27
  request.meta["proxy_headers"] = {"X-ProxyMesh-Country": "US"}
22
28
  ```
23
29
 
24
- Currently this package does not yet support reading custom response headers from the proxy, but that feature is coming soon.
30
+ Any response headers that might come from the proxy will be saved in `response.headers`, as in `response.headers["X-ProxyMesh-IP"]`.
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
4
4
 
5
5
  [project]
6
6
  name = "scrapy-proxy-headers"
7
- version = "0.1.2"
7
+ version = "0.1.4"
8
8
  authors = [
9
9
  { name="ProxyMesh", email="support@proxymesh.com" },
10
10
  ]
@@ -23,3 +23,4 @@ classifiers = [
23
23
  [project.urls]
24
24
  Homepage = "https://github.com/proxymesh/scrapy-proxy-headers"
25
25
  Changelog = "https://github.com/proxymesh/scrapy-proxy-headers/commits/main/"
26
+ Issues = "https://github.com/proxymesh/scrapy-proxy-headers/issues"
@@ -1,6 +1,8 @@
1
- from scrapy.core.downloader.handlers.http11 import TunnelingAgent, TunnelingTCP4ClientEndpoint, ScrapyAgent, HTTP11DownloadHandler
1
+ import re
2
+ from scrapy.core.downloader.handlers.http11 import TunnelingAgent, TunnelingTCP4ClientEndpoint, ScrapyAgent
2
3
  from scrapy.core.downloader.webclient import _parse
3
4
  from scrapy.utils.python import to_bytes
5
+ from scrapy.http import Headers, Response
4
6
 
5
7
  def tunnel_request_data_with_headers(host: str, port: int, **proxy_headers) -> bytes:
6
8
  r"""
@@ -54,6 +56,19 @@ class TunnelingHeadersTCP4ClientEndpoint(TunnelingTCP4ClientEndpoint):
54
56
  protocol.dataReceived = self.processProxyResponse # type: ignore[method-assign]
55
57
  self._protocol = protocol
56
58
  return protocol
59
+
60
+ def processProxyResponse(self, data: bytes):
61
+ # data might have proxy headers, looks like
62
+ # b'HTTP/1.1 200 Connection established\r\nProxy-Header: VALUE\r\n\r\n'
63
+ response_headers = {}
64
+
65
+ for line in data.split(b'\r\n'):
66
+ if b':' in line:
67
+ key, val = line.split(b':', 1)
68
+ response_headers[key.strip()] = val.strip()
69
+ # save for endpoing & agent
70
+ self._proxy_response_headers = Headers(response_headers)
71
+ return super(TunnelingHeadersTCP4ClientEndpoint, self).processProxyResponse(data)
57
72
 
58
73
  class TunnelingHeadersAgent(TunnelingAgent):
59
74
  """An agent that uses a L{TunnelingTCP4ClientEndpoint} to make HTTPS
@@ -70,7 +85,8 @@ class TunnelingHeadersAgent(TunnelingAgent):
70
85
  self._proxy_headers = proxy_headers
71
86
 
72
87
  def _getEndpoint(self, uri):
73
- return TunnelingHeadersTCP4ClientEndpoint(
88
+ # save endpoint for agent to get proxy_response_headers
89
+ self._endpoint = TunnelingHeadersTCP4ClientEndpoint(
74
90
  reactor=self._reactor,
75
91
  host=uri.host,
76
92
  port=uri.port,
@@ -80,31 +96,34 @@ class TunnelingHeadersAgent(TunnelingAgent):
80
96
  bindAddress=self._endpointFactory._bindAddress,
81
97
  **self._proxy_headers
82
98
  )
99
+ return self._endpoint
83
100
 
84
101
  class ScrapyProxyHeadersAgent(ScrapyAgent):
85
102
  _TunnelingAgent = TunnelingHeadersAgent
103
+
104
+ def __init__(self, *args, **kwargs):
105
+ super().__init__(*args, **kwargs)
106
+ self._agent = None
86
107
 
87
108
  def _get_agent(self, request, timeout: float):
88
- agent = super()._get_agent(request, timeout)
109
+ self._agent = super()._get_agent(request, timeout)
89
110
 
90
111
  proxy = request.meta.get("proxy")
91
112
  proxy_headers = request.meta.get('proxy_headers')
92
113
  if proxy and proxy_headers:
93
114
  scheme = _parse(request.url)[0]
94
115
  if scheme == b"https":
95
- agent.set_proxy_headers(proxy_headers)
116
+ self._agent.set_proxy_headers(proxy_headers)
96
117
 
97
- return agent
118
+ return self._agent
98
119
 
99
- class HTTP11ProxyDownloadHandler(HTTP11DownloadHandler):
100
- def download_request(self, request, spider):
101
- """Return a deferred for the HTTP download"""
102
- agent = ScrapyProxyHeadersAgent(
103
- contextFactory=self._contextFactory,
104
- pool=self._pool,
105
- maxsize=getattr(spider, "download_maxsize", self._default_maxsize),
106
- warnsize=getattr(spider, "download_warnsize", self._default_warnsize),
107
- fail_on_dataloss=self._fail_on_dataloss,
108
- crawler=self._crawler,
109
- )
110
- return agent.download_request(request)
120
+ def _cb_bodydone(self, result, request, url: str):
121
+ r = super()._cb_bodydone(result, request, url)
122
+ if isinstance(r, Response):
123
+ if self._agent and hasattr(self._agent, '_endpoint'):
124
+ proxy_response_headers = getattr(self._agent._endpoint, '_proxy_response_headers', None)
125
+ if proxy_response_headers:
126
+ r.headers.update(proxy_response_headers)
127
+ # save this for download handler
128
+ r._proxy_response_headers = proxy_response_headers
129
+ return r
@@ -0,0 +1,37 @@
1
+ from scrapy.core.downloader.handlers.http11 import HTTP11DownloadHandler
2
+ from scrapy_proxy_headers.agent import ScrapyProxyHeadersAgent
3
+
4
+ class HTTP11ProxyDownloadHandler(HTTP11DownloadHandler):
5
+ def __init__(self, *args, **kwargs):
6
+ super().__init__(*args, **kwargs)
7
+ self._proxy_headers_by_proxy = {}
8
+
9
+ def download_request(self, request, spider):
10
+ """Return a deferred for the HTTP download"""
11
+ agent = ScrapyProxyHeadersAgent(
12
+ contextFactory=self._contextFactory,
13
+ pool=self._pool,
14
+ maxsize=getattr(spider, "download_maxsize", self._default_maxsize),
15
+ warnsize=getattr(spider, "download_warnsize", self._default_warnsize),
16
+ fail_on_dataloss=self._fail_on_dataloss,
17
+ crawler=self._crawler,
18
+ )
19
+ response = agent.download_request(request)
20
+ proxy = request.meta.get("proxy")
21
+
22
+ if proxy:
23
+ # we need to do all this because the proxy tunnels can get re-used
24
+ # when that happens, the proxy headers are not available in subsequent responses
25
+ # so we need to save the proxy headers by the proxy, from the first tunnel response
26
+ # so we can add them to subsequent responses
27
+ def callback(response):
28
+ if hasattr(response, '_proxy_response_headers'):
29
+ self._proxy_headers_by_proxy[proxy] = response._proxy_response_headers
30
+
31
+ if proxy in self._proxy_headers_by_proxy:
32
+ response.headers.update(self._proxy_headers_by_proxy[proxy])
33
+
34
+ return response
35
+
36
+ response.addCallback(callback)
37
+ return response
@@ -1,10 +1,11 @@
1
- Metadata-Version: 2.2
1
+ Metadata-Version: 2.1
2
2
  Name: scrapy-proxy-headers
3
- Version: 0.1.2
3
+ Version: 0.1.4
4
4
  Summary: Add custom proxy headers to HTTPS requests in Scrapy
5
5
  Author-email: ProxyMesh <support@proxymesh.com>
6
6
  Project-URL: Homepage, https://github.com/proxymesh/scrapy-proxy-headers
7
7
  Project-URL: Changelog, https://github.com/proxymesh/scrapy-proxy-headers/commits/main/
8
+ Project-URL: Issues, https://github.com/proxymesh/scrapy-proxy-headers/issues
8
9
  Classifier: Programming Language :: Python :: 3
9
10
  Classifier: Operating System :: OS Independent
10
11
  Classifier: License :: OSI Approved :: BSD License
@@ -15,16 +16,20 @@ Requires-Python: >=3.8
15
16
  Description-Content-Type: text/markdown
16
17
  License-File: LICENSE
17
18
 
18
- The `scrapy-proxy-headers` package is designed for adding proxy headers to HTTPS requests.
19
+ # Scrapy Proxy Headers
20
+
21
+ The `scrapy-proxy-headers` package is designed for adding proxy headers to HTTPS requests in [Scrapy](https://scrapy.org/).
19
22
 
20
23
  In normal usage, custom headers put in `request.headers` cannot be read by a proxy when you make a HTTPS request, because the headers are encrypted and passed through the proxy tunnel, along with the rest of the request body. You can read more about this at [Proxy Server Requests over HTTPS](https://docs.proxymesh.com/article/145-proxy-server-requests-over-https).
21
24
 
22
- Because Scrapy does not have a good way to pass custom headers to a proxy when you make HTTPS requests, we at [ProxyMesh](https://proxymesh.com) made this extension to support our customers that use Scrapy and want to use custom headers to control our proxy behavior. But this extension can work for any custom headers through a proxy.
25
+ Because Scrapy does not have a good way to pass custom headers to a proxy when you make HTTPS requests, we at [ProxyMesh](https://proxymesh.com) made this extension to support our customers that use Scrapy and want to use custom headers to control our proxy behavior. But this extension can work for handling custom headers through any proxy.
26
+
27
+ ## Installation
23
28
 
24
29
  To use this extension, do the following:
25
30
 
26
- 1. `pip install scrapy_proxy_headers`
27
- 2. In your Scrapy `settings.py`, add the following code:
31
+ 1. `pip install scrapy-proxy-headers`
32
+ 2. In your Scrapy `settings.py`, add the following:
28
33
 
29
34
  ```python
30
35
  DOWNLOAD_HANDLERS = {
@@ -32,10 +37,12 @@ DOWNLOAD_HANDLERS = {
32
37
  }
33
38
  ```
34
39
 
35
- 3. When you want make a request with a custom proxy header, instead of using `request.headers`, use `request.meta["proxy_headers"]` like this:
40
+ ## Usage
41
+
42
+ When you want make a request with a custom proxy header, instead of using `request.headers`, use `request.meta["proxy_headers"]` like this:
36
43
 
37
44
  ```python
38
45
  request.meta["proxy_headers"] = {"X-ProxyMesh-Country": "US"}
39
46
  ```
40
47
 
41
- Currently this package does not yet support reading custom response headers from the proxy, but that feature is coming soon.
48
+ Any response headers that might come from the proxy will be saved in `response.headers`, as in `response.headers["X-ProxyMesh-IP"]`.
@@ -1,15 +0,0 @@
1
- from scrapy.core.downloader.handlers.http11 import HTTP11DownloadHandler
2
- from scrapy_proxy_headers.agent import ScrapyProxyHeadersAgent
3
-
4
- class HTTP11ProxyDownloadHandler(HTTP11DownloadHandler):
5
- def download_request(self, request, spider):
6
- """Return a deferred for the HTTP download"""
7
- agent = ScrapyProxyHeadersAgent(
8
- contextFactory=self._contextFactory,
9
- pool=self._pool,
10
- maxsize=getattr(spider, "download_maxsize", self._default_maxsize),
11
- warnsize=getattr(spider, "download_warnsize", self._default_warnsize),
12
- fail_on_dataloss=self._fail_on_dataloss,
13
- crawler=self._crawler,
14
- )
15
- return agent.download_request(request)