requests_ruby 1.0.3 → 1.0.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +31 -3
- data/README.md +61 -14
- data/lib/requests/adapters.rb +15 -1
- data/lib/requests/api.rb +9 -2
- data/lib/requests/hpack.rb +241 -0
- data/lib/requests/http2.rb +469 -0
- data/lib/requests/sessions.rb +34 -10
- data/lib/requests/utils.rb +18 -0
- data/lib/requests/version.rb +1 -1
- data/lib/requests.rb +2 -0
- metadata +9 -3
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 8b4146e6fbead1e412259712bcd0939b7971d542bd94a41007e210c9c51de37c
|
|
4
|
+
data.tar.gz: bbc39cd987b00d948be87f9b02251d2e058016123c6ad2fda16724d921575f24
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 658ca79dbadae76f08300daa6ef77b5302bf8c8b2a42877659b490ffdc8359cc07f4c61ff79b56e181f3a4173d9474f5447170420198ee3bb8ded744404f58d1
|
|
7
|
+
data.tar.gz: 37e529b01d2a6406e86332e1ce10835c1b124b7076ceba696da237c215a0e3dab48a570e526dd75b614a95bce426bd1ce87c2a76035867d78fd45f33fa2fef5e
|
data/CHANGELOG.md
CHANGED
|
@@ -1,7 +1,36 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
3
|
+
## 1.0.4
|
|
4
|
+
|
|
5
|
+
Adds real HTTP/2 support and rounds off a handful of gaps that showed up
|
|
6
|
+
against Python's `requests` (and `requests`+`urllib3`) feature set.
|
|
7
|
+
|
|
8
|
+
### Added
|
|
9
|
+
- **HTTP/2 support**, implemented from scratch on top of stdlib
|
|
10
|
+
`Socket`/`OpenSSL` (ALPN negotiation, HPACK header compression with
|
|
11
|
+
Huffman coding, frame multiplexing, flow control) - no external gems.
|
|
12
|
+
`Requests::Http2Adapter` is a drop-in `HTTPAdapter` replacement:
|
|
13
|
+
`Requests::Session.new(http2: true)`, `session.http2!`, or per-call
|
|
14
|
+
`http2: true` on any request/`download`. Transparently falls back to
|
|
15
|
+
HTTP/1.1 for plain `http://` URLs, proxied requests, or servers that
|
|
16
|
+
don't negotiate `h2` over ALPN.
|
|
17
|
+
- `Session#trust_env` (default `true`) - proxies are now picked up from
|
|
18
|
+
`HTTP_PROXY`/`HTTPS_PROXY`/`NO_PROXY` (and lowercase variants) when no
|
|
19
|
+
explicit `proxies:` is set, matching python `requests`' default
|
|
20
|
+
behavior. Set `session.trust_env = false` to opt out.
|
|
21
|
+
- Retry backoff now honors a server's `Retry-After` header (seconds or
|
|
22
|
+
HTTP-date) when retrying a status in `status_forcelist`, instead of
|
|
23
|
+
always using the fixed `backoff_factor * attempt` formula.
|
|
24
|
+
- `Session#head!` / `#options!` and module-level `Requests.get!` /
|
|
25
|
+
`#post!` / `#put!` / `#patch!` / `#delete!` / `#head!` / `#options!` -
|
|
26
|
+
the raise-on-error helpers now cover every verb at both levels.
|
|
27
|
+
- `Requests.session(**opts)` now forwards keyword args (e.g.
|
|
28
|
+
`Requests.session(http2: true)`) instead of always building a bare
|
|
29
|
+
`Session.new`.
|
|
30
|
+
|
|
31
|
+
### Fixed
|
|
32
|
+
- Resolved a leftover unresolved git merge conflict in this file.
|
|
33
|
+
|
|
5
34
|
## 1.0.3
|
|
6
35
|
|
|
7
36
|
Focused on the two things people actually hit walls on with a stdlib-only
|
|
@@ -40,7 +69,6 @@ behave like cookies instead of a flat global hash.
|
|
|
40
69
|
`backoff_factor`, or `status_forcelist` no longer gets silently
|
|
41
70
|
overwritten back to the session's defaults on the next request.
|
|
42
71
|
|
|
43
|
-
>>>>>>> cf89f78 (v1.0.3)
|
|
44
72
|
## 1.0.2
|
|
45
73
|
|
|
46
74
|
Big cleanup pass. Split the single `lib/requests.rb` file into a proper
|
data/README.md
CHANGED
|
@@ -4,6 +4,7 @@ A zero-dependency, stdlib-only HTTP client for Ruby, modeled closely on Python's
|
|
|
4
4
|
|
|
5
5
|
[](https://rubygems.org/gems/requests_ruby)
|
|
6
6
|
[](https://www.ruby-lang.org)
|
|
7
|
+
[](https://rubygems.org/gems/requests_ruby)
|
|
7
8
|
[](LICENSE)
|
|
8
9
|
[](requests_ruby.gemspec)
|
|
9
10
|
|
|
@@ -29,6 +30,7 @@ A zero-dependency, stdlib-only HTTP client for Ruby, modeled closely on Python's
|
|
|
29
30
|
- [Streaming & downloads](#streaming--downloads)
|
|
30
31
|
- [Downloading big files in chunks](#downloading-big-files-in-chunks)
|
|
31
32
|
- [Connection pooling](#connection-pooling)
|
|
33
|
+
- [HTTP/2](#http2)
|
|
32
34
|
- [Custom transport adapters](#custom-transport-adapters)
|
|
33
35
|
- [Status codes](#status-codes)
|
|
34
36
|
- [API reference](#-api-reference)
|
|
@@ -221,13 +223,18 @@ Requests.get(url, timeout: [3, 10]) # [connect, read]
|
|
|
221
223
|
# retry transient connection errors on idempotent methods (GET/HEAD/OPTIONS/PUT/DELETE)
|
|
222
224
|
s = Requests::Session.new
|
|
223
225
|
s.retries = 3
|
|
224
|
-
s.backoff_factor = 0.5 # sleeps 0.5s, 1s, 1.5s between attempts
|
|
226
|
+
s.backoff_factor = 0.5 # sleeps 0.5s, 1s, 1.5s between attempts, unless...
|
|
227
|
+
s.status_forcelist = [429, 502, 503, 504] # ...also retry these response codes
|
|
225
228
|
s.get(url)
|
|
226
229
|
|
|
227
230
|
# or per-request
|
|
228
231
|
Requests.get(url, retries: 3)
|
|
229
232
|
```
|
|
230
233
|
|
|
234
|
+
A response in `status_forcelist` that comes back with a `Retry-After`
|
|
235
|
+
header (seconds or an HTTP date) is retried after that delay instead of
|
|
236
|
+
the `backoff_factor` formula.
|
|
237
|
+
|
|
231
238
|
### Proxies & SSL
|
|
232
239
|
|
|
233
240
|
```ruby
|
|
@@ -236,6 +243,12 @@ Requests.get(url, verify: false) # skip cert verification (careful
|
|
|
236
243
|
Requests.get(url, verify: '/path/to/ca.pem') # custom CA bundle
|
|
237
244
|
```
|
|
238
245
|
|
|
246
|
+
By default (`session.trust_env = true`), a `Session` without an explicit
|
|
247
|
+
`proxies:` picks one up from the `HTTP_PROXY`/`HTTPS_PROXY`/`NO_PROXY`
|
|
248
|
+
environment variables (lowercase names also work), same as python
|
|
249
|
+
`requests`. Set `session.trust_env = false` to ignore the environment
|
|
250
|
+
entirely.
|
|
251
|
+
|
|
239
252
|
A bundled `cacert.pem` (Mozilla's CA bundle) ships with the gem and is used by default. Override it globally with the `REQUESTS_CA_FILE` environment variable.
|
|
240
253
|
|
|
241
254
|
### Cookies
|
|
@@ -332,6 +345,28 @@ it's transparently dropped and reopened on the next request; combined with
|
|
|
332
345
|
`retries`/`backoff_factor` this makes long-lived sessions much more resilient
|
|
333
346
|
than opening a fresh connection per call.
|
|
334
347
|
|
|
348
|
+
### HTTP/2
|
|
349
|
+
|
|
350
|
+
As of 1.0.4, `Requests::Http2Adapter` speaks HTTP/2 directly over
|
|
351
|
+
`Socket`/`OpenSSL` - ALPN negotiation, HPACK header compression, and
|
|
352
|
+
stream multiplexing/flow control, all stdlib, no external gem:
|
|
353
|
+
|
|
354
|
+
```ruby
|
|
355
|
+
s = Requests::Session.new(http2: true)
|
|
356
|
+
s.get('https://api.example.com/a') # negotiates h2 over ALPN, reuses the connection
|
|
357
|
+
s.get('https://api.example.com/b') # same connection, new stream
|
|
358
|
+
|
|
359
|
+
# or opt in per call on a plain session
|
|
360
|
+
Requests.get('https://api.example.com', http2: true)
|
|
361
|
+
resp = Requests::Session.new.tap(&:http2!).get('https://api.example.com')
|
|
362
|
+
```
|
|
363
|
+
|
|
364
|
+
If the server doesn't advertise `h2` during the TLS handshake, or the URL
|
|
365
|
+
is plain `http://`, or a proxy is configured, requests on that host
|
|
366
|
+
transparently fall back to the regular `HTTPAdapter` (HTTP/1.1) - you get
|
|
367
|
+
the same `Response` object either way and don't need to know which
|
|
368
|
+
transport actually served the request.
|
|
369
|
+
|
|
335
370
|
### Custom transport adapters
|
|
336
371
|
|
|
337
372
|
Loosely inspired by Python's `Session.mount()` / `HTTPAdapter`:
|
|
@@ -362,15 +397,16 @@ Requests.codes.im_a_teapot # 418, yes really
|
|
|
362
397
|
|---|---|---|
|
|
363
398
|
| `Requests.get/post/put/patch/delete/head/options` | `session.get/post/put/patch/delete/head/options` | `status_code`, `status` |
|
|
364
399
|
| `Requests.request(method, url, **opts)` | `session.request(method, url, **opts)` | `ok?`, `redirect?` |
|
|
365
|
-
| `Requests.
|
|
366
|
-
| `Requests.
|
|
367
|
-
| `Requests.
|
|
368
|
-
|
|
|
369
|
-
| | `session.
|
|
400
|
+
| `Requests.get!/post!/put!/patch!/delete!/head!/options!` (raise on error) | `session.get!/post!/put!/patch!/delete!/head!/options!` (raise on error) | `text`, `content`, `json` |
|
|
401
|
+
| `Requests.session(**opts)` → new `Session` | `session.mount(prefix, adapter)`, `session.close` | `headers`, `cookies` |
|
|
402
|
+
| `Requests.download(url, to:, chunk_size:, resume:, progress:)` | `session.download(url, to:, ...)` (same, but reuses the session's pool/auth/headers) | `elapsed`, `history`, `url` |
|
|
403
|
+
| `Requests.codes` | `session.http2!`, `Session.new(http2: true)`, or `http2: true` per request | `raise_for_status`, `save_to`, `links` |
|
|
404
|
+
| | `session.hooks[:response]` | |
|
|
405
|
+
| | `session.headers`, `.cookies`, `.auth`, `.proxies`, `.trust_env`, `.retries`, `.backoff_factor`, `.status_forcelist` | |
|
|
370
406
|
| | `Requests::Jar#save(path)` / `Requests::Jar.load(path)` | |
|
|
371
|
-
| | `HTTPAdapter.new(max_retries:, backoff_factor:, status_forcelist:)` | |
|
|
407
|
+
| | `HTTPAdapter.new(max_retries:, backoff_factor:, status_forcelist:)`, `Http2Adapter.new(...)` | |
|
|
372
408
|
|
|
373
|
-
Every request-level method accepts: `params`, `data`, `json`, `headers`, `cookies`, `files`, `auth`, `timeout`, `allow_redirects`, `proxies`, `verify`, `cert`, `hooks`, `retries`.
|
|
409
|
+
Every request-level method accepts: `params`, `data`, `json`, `headers`, `cookies`, `files`, `auth`, `timeout`, `allow_redirects`, `proxies`, `verify`, `cert`, `hooks`, `retries`, `http2`.
|
|
374
410
|
|
|
375
411
|
## Python parity cheat sheet
|
|
376
412
|
|
|
@@ -397,15 +433,22 @@ Every request-level method accepts: `params`, `data`, `json`, `headers`, `cookie
|
|
|
397
433
|
- `iter_content`/`iter_lines` on a regular response read the whole body up
|
|
398
434
|
front rather than lazily streaming it off the socket — `download` (added
|
|
399
435
|
in 1.0.3) is the one that streams for real, use that for big files.
|
|
400
|
-
-
|
|
401
|
-
|
|
436
|
+
- HTTP/2 (1.0.4) only applies to direct `https://` connections - there's
|
|
437
|
+
no `h2c` (cleartext HTTP/2) support, and requests through a proxy always
|
|
438
|
+
use HTTP/1.1. No automatic NTLM/Kerberos/OAuth either — bring your own
|
|
439
|
+
`auth:` object (anything with `#call(headers)` works) if you need those.
|
|
440
|
+
- The HTTP/2 adapter applies `timeout:` as one coarse deadline around the
|
|
441
|
+
whole request/response exchange rather than resetting it on every
|
|
442
|
+
individual socket read the way the HTTP/1.1 adapter's `Net::HTTP`
|
|
443
|
+
read-timeout does.
|
|
402
444
|
- `verify: '/path/to/ca.pem'` accepts a single bundle file, not a
|
|
403
445
|
directory of certs (`Net::HTTP` itself only takes `ca_file` or
|
|
404
446
|
`ca_path`, and this gem only wires up the former today).
|
|
405
|
-
- The connection pool added in 1.0.3 is per
|
|
406
|
-
instance and isn't thread-safe for concurrent
|
|
407
|
-
`Session` object — use one `Session` per thread,
|
|
408
|
-
thread, same advice as most connection-pooling
|
|
447
|
+
- The connection pool added in 1.0.3 (HTTP/1.1 and HTTP/2 alike) is per-
|
|
448
|
+
`Session`/per-adapter instance and isn't thread-safe for concurrent
|
|
449
|
+
requests on the *same* `Session` object — use one `Session` per thread,
|
|
450
|
+
or one connection per thread, same advice as most connection-pooling
|
|
451
|
+
HTTP clients.
|
|
409
452
|
|
|
410
453
|
|
|
411
454
|
## 🔧 Development
|
|
@@ -428,6 +471,10 @@ Bug reports and pull requests are welcome. A few ground rules:
|
|
|
428
471
|
2. Add a test in `spec/requests_spec.rb` for anything you fix or add.
|
|
429
472
|
3. Keep the Python-parity naming where it makes sense, but don't force it where it doesn't fit Ruby idiom (`?`-suffixed predicates, etc.).
|
|
430
473
|
|
|
474
|
+
## Support the project
|
|
475
|
+
|
|
476
|
+
If requests_ruby helped you, consider starring the repository ⭐
|
|
477
|
+
|
|
431
478
|
## License
|
|
432
479
|
|
|
433
480
|
[MIT](LICENSE)
|
data/lib/requests/adapters.rb
CHANGED
|
@@ -5,6 +5,7 @@ require 'openssl'
|
|
|
5
5
|
require 'uri'
|
|
6
6
|
require 'zlib'
|
|
7
7
|
require 'stringio'
|
|
8
|
+
require 'time'
|
|
8
9
|
|
|
9
10
|
module Requests
|
|
10
11
|
class HTTPAdapter
|
|
@@ -30,7 +31,7 @@ module Requests
|
|
|
30
31
|
r = do_send(meth, url, hdrs, body, timeout, proxies, verify, cert, auth)
|
|
31
32
|
if @status_forcelist.include?(r.code.to_i) && attempt < @max_retries
|
|
32
33
|
attempt += 1
|
|
33
|
-
sleep(
|
|
34
|
+
sleep(retry_delay(r, attempt))
|
|
34
35
|
next
|
|
35
36
|
end
|
|
36
37
|
return r
|
|
@@ -104,6 +105,19 @@ module Requests
|
|
|
104
105
|
def idempotent?(meth)
|
|
105
106
|
%w[GET HEAD OPTIONS PUT DELETE].include?(meth.to_s.upcase)
|
|
106
107
|
end
|
|
108
|
+
def retry_delay(resp, attempt)
|
|
109
|
+
ra = resp['retry-after']
|
|
110
|
+
if ra
|
|
111
|
+
return ra.to_i if ra =~ /\A\d+\z/
|
|
112
|
+
begin
|
|
113
|
+
d = Time.httpdate(ra) - Time.now
|
|
114
|
+
return d.positive? ? d : 0
|
|
115
|
+
rescue ArgumentError
|
|
116
|
+
nil
|
|
117
|
+
end
|
|
118
|
+
end
|
|
119
|
+
@backoff_factor.to_f > 0 ? @backoff_factor * attempt : 0
|
|
120
|
+
end
|
|
107
121
|
def do_send(meth, url, hdrs, body, timeout, proxies, verify, cert, auth)
|
|
108
122
|
u = URI.parse(url)
|
|
109
123
|
auth.call(hdrs) if auth.respond_to?(:call) && !auth.is_a?(Requests::DigestAuth)
|
data/lib/requests/api.rb
CHANGED
|
@@ -14,8 +14,15 @@ module Requests
|
|
|
14
14
|
kw[:allow_redirects] = kw.fetch(:allow_redirects, false)
|
|
15
15
|
request('HEAD', url, **kw)
|
|
16
16
|
end
|
|
17
|
-
def self.
|
|
18
|
-
|
|
17
|
+
def self.get!(url, **kw); get(url, **kw).raise_for_status; end
|
|
18
|
+
def self.post!(url, **kw); post(url, **kw).raise_for_status; end
|
|
19
|
+
def self.put!(url, **kw); put(url, **kw).raise_for_status; end
|
|
20
|
+
def self.patch!(url, **kw); patch(url, **kw).raise_for_status; end
|
|
21
|
+
def self.delete!(url, **kw); delete(url, **kw).raise_for_status; end
|
|
22
|
+
def self.head!(url, **kw); head(url, **kw).raise_for_status; end
|
|
23
|
+
def self.options!(url, **kw); options(url, **kw).raise_for_status; end
|
|
24
|
+
def self.session(**kw)
|
|
25
|
+
Session.new(**kw)
|
|
19
26
|
end
|
|
20
27
|
def self.download(url, to:, **kw)
|
|
21
28
|
Session.new.download(url, to: to, **kw)
|
|
@@ -0,0 +1,241 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Requests
|
|
4
|
+
module HPACK
|
|
5
|
+
STATIC_TABLE = [
|
|
6
|
+
[':authority', ''],[':method', 'GET'],[':method', 'POST'],[':path', '/'],
|
|
7
|
+
[':path', '/index.html'],[':scheme', 'http'],[':scheme', 'https'],[':status', '200'],
|
|
8
|
+
[':status', '204'],[':status', '206'],[':status', '304'],[':status', '400'],
|
|
9
|
+
[':status', '404'],[':status', '500'],['accept-charset', ''],['accept-encoding', 'gzip, deflate'],
|
|
10
|
+
['accept-language', ''],['accept-ranges', ''],['accept', ''],['access-control-allow-origin', ''],
|
|
11
|
+
['age', ''],['allow', ''],['authorization', ''],['cache-control', ''],
|
|
12
|
+
['content-disposition', ''],['content-encoding', ''],['content-language', ''],['content-length', ''],
|
|
13
|
+
['content-location', ''],['content-range', ''],['content-type', ''],['cookie', ''],
|
|
14
|
+
['date', ''],['etag', ''],['expect', ''],['expires', ''],
|
|
15
|
+
['from', ''],['host', ''],['if-match', ''],['if-modified-since', ''],
|
|
16
|
+
['if-none-match', ''],['if-range', ''],['if-unmodified-since', ''],['last-modified', ''],
|
|
17
|
+
['link', ''],['location', ''],['max-forwards', ''],['proxy-authenticate', ''],
|
|
18
|
+
['proxy-authorization', ''],['range', ''],['referer', ''],['refresh', ''],
|
|
19
|
+
['retry-after', ''],['server', ''],['set-cookie', ''],['strict-transport-security', ''],
|
|
20
|
+
['transfer-encoding', ''],['user-agent', ''],['vary', ''],['via', ''],
|
|
21
|
+
['www-authenticate', '']
|
|
22
|
+
].freeze
|
|
23
|
+
|
|
24
|
+
module Huffman
|
|
25
|
+
CODES = [
|
|
26
|
+
[0x1ff8, 13],[0x7fffd8, 23],[0xfffffe2, 28],[0xfffffe3, 28],[0xfffffe4, 28],[0xfffffe5, 28],
|
|
27
|
+
[0xfffffe6, 28],[0xfffffe7, 28],[0xfffffe8, 28],[0xffffea, 24],[0x3ffffffc, 30],[0xfffffe9, 28],
|
|
28
|
+
[0xfffffea, 28],[0x3ffffffd, 30],[0xfffffeb, 28],[0xfffffec, 28],[0xfffffed, 28],[0xfffffee, 28],
|
|
29
|
+
[0xfffffef, 28],[0xffffff0, 28],[0xffffff1, 28],[0xffffff2, 28],[0x3ffffffe, 30],[0xffffff3, 28],
|
|
30
|
+
[0xffffff4, 28],[0xffffff5, 28],[0xffffff6, 28],[0xffffff7, 28],[0xffffff8, 28],[0xffffff9, 28],
|
|
31
|
+
[0xffffffa, 28],[0xffffffb, 28],[0x14, 6],[0x3f8, 10],[0x3f9, 10],[0xffa, 12],
|
|
32
|
+
[0x1ff9, 13],[0x15, 6],[0xf8, 8],[0x7fa, 11],[0x3fa, 10],[0x3fb, 10],
|
|
33
|
+
[0xf9, 8],[0x7fb, 11],[0xfa, 8],[0x16, 6],[0x17, 6],[0x18, 6],
|
|
34
|
+
[0x0, 5],[0x1, 5],[0x2, 5],[0x19, 6],[0x1a, 6],[0x1b, 6],
|
|
35
|
+
[0x1c, 6],[0x1d, 6],[0x1e, 6],[0x1f, 6],[0x5c, 7],[0xfb, 8],
|
|
36
|
+
[0x7ffc, 15],[0x20, 6],[0xffb, 12],[0x3fc, 10],[0x1ffa, 13],[0x21, 6],
|
|
37
|
+
[0x5d, 7],[0x5e, 7],[0x5f, 7],[0x60, 7],[0x61, 7],[0x62, 7],
|
|
38
|
+
[0x63, 7],[0x64, 7],[0x65, 7],[0x66, 7],[0x67, 7],[0x68, 7],
|
|
39
|
+
[0x69, 7],[0x6a, 7],[0x6b, 7],[0x6c, 7],[0x6d, 7],[0x6e, 7],
|
|
40
|
+
[0x6f, 7],[0x70, 7],[0x71, 7],[0x72, 7],[0xfc, 8],[0x73, 7],
|
|
41
|
+
[0xfd, 8],[0x1ffb, 13],[0x7fff0, 19],[0x1ffc, 13],[0x3ffc, 14],[0x22, 6],
|
|
42
|
+
[0x7ffd, 15],[0x3, 5],[0x23, 6],[0x4, 5],[0x24, 6],[0x5, 5],
|
|
43
|
+
[0x25, 6],[0x26, 6],[0x27, 6],[0x6, 5],[0x74, 7],[0x75, 7],
|
|
44
|
+
[0x28, 6],[0x29, 6],[0x2a, 6],[0x7, 5],[0x2b, 6],[0x76, 7],
|
|
45
|
+
[0x2c, 6],[0x8, 5],[0x9, 5],[0x2d, 6],[0x77, 7],[0x78, 7],
|
|
46
|
+
[0x79, 7],[0x7a, 7],[0x7b, 7],[0x7ffe, 15],[0x7fc, 11],[0x3ffd, 14],
|
|
47
|
+
[0x1ffd, 13],[0xffffffc, 28],[0xfffe6, 20],[0x3fffd2, 22],[0xfffe7, 20],[0xfffe8, 20],
|
|
48
|
+
[0x3fffd3, 22],[0x3fffd4, 22],[0x3fffd5, 22],[0x7fffd9, 23],[0x3fffd6, 22],[0x7fffda, 23],
|
|
49
|
+
[0x7fffdb, 23],[0x7fffdc, 23],[0x7fffdd, 23],[0x7fffde, 23],[0xffffeb, 24],[0x7fffdf, 23],
|
|
50
|
+
[0xffffec, 24],[0xffffed, 24],[0x3fffd7, 22],[0x7fffe0, 23],[0xffffee, 24],[0x7fffe1, 23],
|
|
51
|
+
[0x7fffe2, 23],[0x7fffe3, 23],[0x7fffe4, 23],[0x1fffdc, 21],[0x3fffd8, 22],[0x7fffe5, 23],
|
|
52
|
+
[0x3fffd9, 22],[0x7fffe6, 23],[0x7fffe7, 23],[0xffffef, 24],[0x3fffda, 22],[0x1fffdd, 21],
|
|
53
|
+
[0xfffe9, 20],[0x3fffdb, 22],[0x3fffdc, 22],[0x7fffe8, 23],[0x7fffe9, 23],[0x1fffde, 21],
|
|
54
|
+
[0x7fffea, 23],[0x3fffdd, 22],[0x3fffde, 22],[0xfffff0, 24],[0x1fffdf, 21],[0x3fffdf, 22],
|
|
55
|
+
[0x7fffeb, 23],[0x7fffec, 23],[0x1fffe0, 21],[0x1fffe1, 21],[0x3fffe0, 22],[0x1fffe2, 21],
|
|
56
|
+
[0x7fffed, 23],[0x3fffe1, 22],[0x7fffee, 23],[0x7fffef, 23],[0xfffea, 20],[0x3fffe2, 22],
|
|
57
|
+
[0x3fffe3, 22],[0x3fffe4, 22],[0x7ffff0, 23],[0x3fffe5, 22],[0x3fffe6, 22],[0x7ffff1, 23],
|
|
58
|
+
[0x3ffffe0, 26],[0x3ffffe1, 26],[0xfffeb, 20],[0x7fff1, 19],[0x3fffe7, 22],[0x7ffff2, 23],
|
|
59
|
+
[0x3fffe8, 22],[0x1ffffec, 25],[0x3ffffe2, 26],[0x3ffffe3, 26],[0x3ffffe4, 26],[0x7ffffde, 27],
|
|
60
|
+
[0x7ffffdf, 27],[0x3ffffe5, 26],[0xfffff1, 24],[0x1ffffed, 25],[0x7fff2, 19],[0x1fffe3, 21],
|
|
61
|
+
[0x3ffffe6, 26],[0x7ffffe0, 27],[0x7ffffe1, 27],[0x3ffffe7, 26],[0x7ffffe2, 27],[0xfffff2, 24],
|
|
62
|
+
[0x1fffe4, 21],[0x1fffe5, 21],[0x3ffffe8, 26],[0x3ffffe9, 26],[0xffffffd, 28],[0x7ffffe3, 27],
|
|
63
|
+
[0x7ffffe4, 27],[0x7ffffe5, 27],[0xfffec, 20],[0xfffff3, 24],[0xfffed, 20],[0x1fffe6, 21],
|
|
64
|
+
[0x3fffe9, 22],[0x1fffe7, 21],[0x1fffe8, 21],[0x7ffff3, 23],[0x3fffea, 22],[0x3fffeb, 22],
|
|
65
|
+
[0x1ffffee, 25],[0x1ffffef, 25],[0xfffff4, 24],[0xfffff5, 24],[0x3ffffea, 26],[0x7ffff4, 23],
|
|
66
|
+
[0x3ffffeb, 26],[0x7ffffe6, 27],[0x3ffffec, 26],[0x3ffffed, 26],[0x7ffffe7, 27],[0x7ffffe8, 27],
|
|
67
|
+
[0x7ffffe9, 27],[0x7ffffea, 27],[0x7ffffeb, 27],[0xffffffe, 28],[0x7ffffec, 27],[0x7ffffed, 27],
|
|
68
|
+
[0x7ffffee, 27],[0x7ffffef, 27],[0x7fffff0, 27],[0x3ffffee, 26],[0x3fffffff, 30]
|
|
69
|
+
].freeze
|
|
70
|
+
EOS = 256
|
|
71
|
+
ENCODE_TABLE = CODES.map { |c, l| [c].pack('N').unpack1('B*')[-l..-1] }.freeze
|
|
72
|
+
def self.build_tree
|
|
73
|
+
root = {}
|
|
74
|
+
CODES.each_with_index do |(code, len), sym|
|
|
75
|
+
node = root
|
|
76
|
+
(len - 1).downto(0) do |i|
|
|
77
|
+
bit = (code >> i) & 1
|
|
78
|
+
node = (node[bit] ||= {})
|
|
79
|
+
end
|
|
80
|
+
node[:sym] = sym
|
|
81
|
+
end
|
|
82
|
+
root
|
|
83
|
+
end
|
|
84
|
+
DECODE_TREE = build_tree
|
|
85
|
+
def self.encode(str)
|
|
86
|
+
bits = String.new
|
|
87
|
+
str.each_byte { |b| bits << ENCODE_TABLE[b] }
|
|
88
|
+
pad = (8 - bits.size % 8) % 8
|
|
89
|
+
bits << ('1' * pad) if pad > 0
|
|
90
|
+
[bits].pack('B*')
|
|
91
|
+
end
|
|
92
|
+
def self.decode(bytes)
|
|
93
|
+
out = String.new(encoding: Encoding::BINARY)
|
|
94
|
+
node = DECODE_TREE
|
|
95
|
+
bitcount = bytes.bytesize * 8
|
|
96
|
+
pos = 0
|
|
97
|
+
bytes.each_byte do |byte|
|
|
98
|
+
7.downto(0) do |i|
|
|
99
|
+
pos += 1
|
|
100
|
+
bit = (byte >> i) & 1
|
|
101
|
+
nxt = node[bit]
|
|
102
|
+
unless nxt
|
|
103
|
+
raise Requests::ContentDecodingError, 'invalid huffman padding' if pos > bitcount - 8 || pos < bitcount
|
|
104
|
+
return out
|
|
105
|
+
end
|
|
106
|
+
node = nxt
|
|
107
|
+
if node.key?(:sym)
|
|
108
|
+
sym = node[:sym]
|
|
109
|
+
raise Requests::ContentDecodingError, 'unexpected huffman EOS' if sym == EOS
|
|
110
|
+
out << sym.chr(Encoding::BINARY)
|
|
111
|
+
node = DECODE_TREE
|
|
112
|
+
end
|
|
113
|
+
end
|
|
114
|
+
end
|
|
115
|
+
out
|
|
116
|
+
end
|
|
117
|
+
end
|
|
118
|
+
|
|
119
|
+
module Integer
|
|
120
|
+
module_function
|
|
121
|
+
def encode(value, prefix_bits, first_byte = 0)
|
|
122
|
+
max = (1 << prefix_bits) - 1
|
|
123
|
+
if value < max
|
|
124
|
+
[first_byte | value].pack('C')
|
|
125
|
+
else
|
|
126
|
+
out = [first_byte | max].pack('C')
|
|
127
|
+
value -= max
|
|
128
|
+
while value >= 128
|
|
129
|
+
out << [(value % 128) | 128].pack('C')
|
|
130
|
+
value /= 128
|
|
131
|
+
end
|
|
132
|
+
out << [value].pack('C')
|
|
133
|
+
end
|
|
134
|
+
end
|
|
135
|
+
def decode(bytes, pos, prefix_bits)
|
|
136
|
+
max = (1 << prefix_bits) - 1
|
|
137
|
+
value = bytes.getbyte(pos) & max
|
|
138
|
+
pos += 1
|
|
139
|
+
if value == max
|
|
140
|
+
m = 0
|
|
141
|
+
loop do
|
|
142
|
+
b = bytes.getbyte(pos)
|
|
143
|
+
raise Requests::ContentDecodingError, 'truncated hpack integer' if b.nil?
|
|
144
|
+
pos += 1
|
|
145
|
+
value += (b & 127) * (2**m)
|
|
146
|
+
m += 7
|
|
147
|
+
break if b & 128 == 0
|
|
148
|
+
end
|
|
149
|
+
end
|
|
150
|
+
[value, pos]
|
|
151
|
+
end
|
|
152
|
+
end
|
|
153
|
+
|
|
154
|
+
class Encoder
|
|
155
|
+
def encode(headers)
|
|
156
|
+
out = String.new(encoding: Encoding::BINARY)
|
|
157
|
+
headers.each { |name, value| out << encode_field(name.to_s.downcase, value.to_s) }
|
|
158
|
+
out
|
|
159
|
+
end
|
|
160
|
+
private
|
|
161
|
+
def encode_field(name, value)
|
|
162
|
+
exact = STATIC_TABLE.index { |n, v| n == name && v == value }
|
|
163
|
+
return Integer.encode(exact + 1, 7, 0x80) if exact
|
|
164
|
+
name_idx = STATIC_TABLE.index { |n, _| n == name }
|
|
165
|
+
head = name_idx ? Integer.encode(name_idx + 1, 4, 0x00) : Integer.encode(0, 4, 0x00) + literal(name)
|
|
166
|
+
head + literal(value)
|
|
167
|
+
end
|
|
168
|
+
def literal(str)
|
|
169
|
+
h = Huffman.encode(str)
|
|
170
|
+
if h.bytesize < str.bytesize
|
|
171
|
+
Integer.encode(h.bytesize, 7, 0x80) + h
|
|
172
|
+
else
|
|
173
|
+
Integer.encode(str.bytesize, 7, 0x00) + str.dup.force_encoding(Encoding::BINARY)
|
|
174
|
+
end
|
|
175
|
+
end
|
|
176
|
+
end
|
|
177
|
+
|
|
178
|
+
class Decoder
|
|
179
|
+
def initialize(max_size: 4096)
|
|
180
|
+
@dynamic = []
|
|
181
|
+
@max_size = max_size
|
|
182
|
+
@size = 0
|
|
183
|
+
end
|
|
184
|
+
def decode(bytes)
|
|
185
|
+
pos = 0
|
|
186
|
+
out = []
|
|
187
|
+
while pos < bytes.bytesize
|
|
188
|
+
byte = bytes.getbyte(pos)
|
|
189
|
+
if byte & 0x80 != 0
|
|
190
|
+
idx, pos = Integer.decode(bytes, pos, 7)
|
|
191
|
+
out << lookup(idx)
|
|
192
|
+
elsif byte & 0x40 != 0
|
|
193
|
+
name, value, pos = read_literal(bytes, pos, 6)
|
|
194
|
+
add_dynamic(name, value)
|
|
195
|
+
out << [name, value]
|
|
196
|
+
elsif byte & 0x20 != 0
|
|
197
|
+
size, pos = Integer.decode(bytes, pos, 5)
|
|
198
|
+
@max_size = size
|
|
199
|
+
evict
|
|
200
|
+
else
|
|
201
|
+
name, value, pos = read_literal(bytes, pos, 4)
|
|
202
|
+
out << [name, value]
|
|
203
|
+
end
|
|
204
|
+
end
|
|
205
|
+
out
|
|
206
|
+
end
|
|
207
|
+
private
|
|
208
|
+
def lookup(idx)
|
|
209
|
+
return STATIC_TABLE[idx - 1] if idx <= STATIC_TABLE.size
|
|
210
|
+
d = @dynamic[idx - STATIC_TABLE.size - 1]
|
|
211
|
+
raise Requests::ContentDecodingError, "invalid hpack index #{idx}" unless d
|
|
212
|
+
d
|
|
213
|
+
end
|
|
214
|
+
def read_literal(bytes, pos, prefix_bits)
|
|
215
|
+
idx, pos = Integer.decode(bytes, pos, prefix_bits)
|
|
216
|
+
name, pos = idx.zero? ? read_string(bytes, pos) : [lookup(idx)[0], pos]
|
|
217
|
+
value, pos = read_string(bytes, pos)
|
|
218
|
+
[name, value, pos]
|
|
219
|
+
end
|
|
220
|
+
def read_string(bytes, pos)
|
|
221
|
+
huff = bytes.getbyte(pos) & 0x80 != 0
|
|
222
|
+
len, pos = Integer.decode(bytes, pos, 7)
|
|
223
|
+
raw = bytes.byteslice(pos, len)
|
|
224
|
+
pos += len
|
|
225
|
+
[huff ? Huffman.decode(raw) : raw, pos]
|
|
226
|
+
end
|
|
227
|
+
def add_dynamic(name, value)
|
|
228
|
+
entry_size = name.bytesize + value.bytesize + 32
|
|
229
|
+
@dynamic.unshift([name, value])
|
|
230
|
+
@size += entry_size
|
|
231
|
+
evict
|
|
232
|
+
end
|
|
233
|
+
def evict
|
|
234
|
+
while @size > @max_size && !@dynamic.empty?
|
|
235
|
+
name, value = @dynamic.pop
|
|
236
|
+
@size -= name.bytesize + value.bytesize + 32
|
|
237
|
+
end
|
|
238
|
+
end
|
|
239
|
+
end
|
|
240
|
+
end
|
|
241
|
+
end
|
|
@@ -0,0 +1,469 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'socket'
|
|
4
|
+
require 'openssl'
|
|
5
|
+
require 'uri'
|
|
6
|
+
require 'zlib'
|
|
7
|
+
require 'timeout'
|
|
8
|
+
require 'time'
|
|
9
|
+
|
|
10
|
+
module Requests
|
|
11
|
+
class Http2Response
|
|
12
|
+
def initialize(header_pairs, body)
|
|
13
|
+
@status = nil
|
|
14
|
+
@fields = Hash.new { |h, k| h[k] = [] }
|
|
15
|
+
@headers = CIHash.new
|
|
16
|
+
header_pairs.each do |k, v|
|
|
17
|
+
if k == ':status'
|
|
18
|
+
@status = v
|
|
19
|
+
else
|
|
20
|
+
@fields[k.downcase] << v
|
|
21
|
+
@headers[k] = v
|
|
22
|
+
end
|
|
23
|
+
end
|
|
24
|
+
@body = body
|
|
25
|
+
end
|
|
26
|
+
def code
|
|
27
|
+
@status
|
|
28
|
+
end
|
|
29
|
+
def message
|
|
30
|
+
(Requests::STATUS_CODES[@status.to_i] || :unknown).to_s.split('_').map(&:capitalize).join(' ')
|
|
31
|
+
end
|
|
32
|
+
def [](k)
|
|
33
|
+
@headers[k]
|
|
34
|
+
end
|
|
35
|
+
def each_header
|
|
36
|
+
return enum_for(:each_header) unless block_given?
|
|
37
|
+
@headers.each { |k, v| yield k, v }
|
|
38
|
+
end
|
|
39
|
+
def get_fields(name)
|
|
40
|
+
f = @fields[name.to_s.downcase]
|
|
41
|
+
f.empty? ? nil : f
|
|
42
|
+
end
|
|
43
|
+
def body
|
|
44
|
+
@body
|
|
45
|
+
end
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
class Http2Connection
|
|
49
|
+
TYPE_DATA = 0x0
|
|
50
|
+
TYPE_HEADERS = 0x1
|
|
51
|
+
TYPE_PRIORITY = 0x2
|
|
52
|
+
TYPE_RST_STREAM = 0x3
|
|
53
|
+
TYPE_SETTINGS = 0x4
|
|
54
|
+
TYPE_PUSH_PROMISE = 0x5
|
|
55
|
+
TYPE_PING = 0x6
|
|
56
|
+
TYPE_GOAWAY = 0x7
|
|
57
|
+
TYPE_WINDOW_UPDATE = 0x8
|
|
58
|
+
TYPE_CONTINUATION = 0x9
|
|
59
|
+
FLAG_END_STREAM = 0x1
|
|
60
|
+
FLAG_ACK = 0x1
|
|
61
|
+
FLAG_END_HEADERS = 0x4
|
|
62
|
+
FLAG_PADDED = 0x8
|
|
63
|
+
FLAG_PRIORITY = 0x20
|
|
64
|
+
PREFACE = "PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n"
|
|
65
|
+
DEFAULT_WINDOW = 65_535
|
|
66
|
+
|
|
67
|
+
def initialize(socket)
|
|
68
|
+
@socket = socket
|
|
69
|
+
@next_stream_id = 1
|
|
70
|
+
@encoder = HPACK::Encoder.new
|
|
71
|
+
@decoder = HPACK::Decoder.new
|
|
72
|
+
@send_window = DEFAULT_WINDOW
|
|
73
|
+
@conn_recv_window = DEFAULT_WINDOW
|
|
74
|
+
@peer_max_frame_size = 16_384
|
|
75
|
+
@peer_initial_window = DEFAULT_WINDOW
|
|
76
|
+
@streams = {}
|
|
77
|
+
@closed = false
|
|
78
|
+
handshake!
|
|
79
|
+
end
|
|
80
|
+
def alive?
|
|
81
|
+
!@closed && !@socket.closed?
|
|
82
|
+
end
|
|
83
|
+
def close
|
|
84
|
+
write_frame(TYPE_GOAWAY, 0, 0, [0, 0].pack('NN')) if alive?
|
|
85
|
+
rescue StandardError
|
|
86
|
+
nil
|
|
87
|
+
ensure
|
|
88
|
+
@closed = true
|
|
89
|
+
@socket.close unless @socket.closed?
|
|
90
|
+
end
|
|
91
|
+
def request(method, uri, header_list, body, io: nil, progress: nil)
|
|
92
|
+
raise Requests::ConnectionError, 'http/2 connection is closed' unless alive?
|
|
93
|
+
stream_id = @next_stream_id
|
|
94
|
+
@next_stream_id += 2
|
|
95
|
+
@streams[stream_id] = { send_window: @peer_initial_window, recv_window: DEFAULT_WINDOW }
|
|
96
|
+
end_stream = body.nil? || body.to_s.empty?
|
|
97
|
+
send_headers(stream_id, header_list, end_stream)
|
|
98
|
+
send_body(stream_id, body) unless end_stream
|
|
99
|
+
read_response(stream_id, io: io, progress: progress)
|
|
100
|
+
ensure
|
|
101
|
+
@streams.delete(stream_id) if stream_id
|
|
102
|
+
end
|
|
103
|
+
private
|
|
104
|
+
def handshake!
|
|
105
|
+
@socket.write(PREFACE)
|
|
106
|
+
write_frame(TYPE_SETTINGS, 0, 0, '')
|
|
107
|
+
loop do
|
|
108
|
+
type, flags, sid, payload = read_frame
|
|
109
|
+
case type
|
|
110
|
+
when TYPE_SETTINGS
|
|
111
|
+
if flags & FLAG_ACK == 0
|
|
112
|
+
apply_settings(payload)
|
|
113
|
+
write_frame(TYPE_SETTINGS, FLAG_ACK, 0, '')
|
|
114
|
+
end
|
|
115
|
+
break
|
|
116
|
+
when TYPE_WINDOW_UPDATE
|
|
117
|
+
apply_window_update(sid, payload)
|
|
118
|
+
end
|
|
119
|
+
end
|
|
120
|
+
end
|
|
121
|
+
def send_headers(stream_id, header_list, end_stream)
|
|
122
|
+
blob = @encoder.encode(header_list)
|
|
123
|
+
offset = 0
|
|
124
|
+
first = true
|
|
125
|
+
loop do
|
|
126
|
+
chunk = blob.byteslice(offset, @peer_max_frame_size) || ''
|
|
127
|
+
offset += chunk.bytesize
|
|
128
|
+
last = offset >= blob.bytesize
|
|
129
|
+
flags = 0
|
|
130
|
+
flags |= FLAG_END_HEADERS if last
|
|
131
|
+
flags |= FLAG_END_STREAM if last && end_stream
|
|
132
|
+
write_frame(first ? TYPE_HEADERS : TYPE_CONTINUATION, flags, stream_id, chunk)
|
|
133
|
+
first = false
|
|
134
|
+
break if last
|
|
135
|
+
end
|
|
136
|
+
end
|
|
137
|
+
def send_body(stream_id, body)
|
|
138
|
+
pos = 0
|
|
139
|
+
st = @streams[stream_id]
|
|
140
|
+
body = body.dup.force_encoding(Encoding::BINARY)
|
|
141
|
+
loop do
|
|
142
|
+
avail = [st[:send_window], @send_window, @peer_max_frame_size].min
|
|
143
|
+
if avail <= 0
|
|
144
|
+
type, flags, sid, payload = read_frame
|
|
145
|
+
dispatch_control_frame(type, flags, sid, payload)
|
|
146
|
+
next
|
|
147
|
+
end
|
|
148
|
+
chunk = body.byteslice(pos, [avail, body.bytesize - pos].min)
|
|
149
|
+
pos += chunk.bytesize
|
|
150
|
+
st[:send_window] -= chunk.bytesize
|
|
151
|
+
@send_window -= chunk.bytesize
|
|
152
|
+
last = pos >= body.bytesize
|
|
153
|
+
write_frame(TYPE_DATA, last ? FLAG_END_STREAM : 0, stream_id, chunk)
|
|
154
|
+
break if last
|
|
155
|
+
end
|
|
156
|
+
end
|
|
157
|
+
def read_response(stream_id, io: nil, progress: nil)
|
|
158
|
+
header_blob = String.new(encoding: Encoding::BINARY)
|
|
159
|
+
headers = nil
|
|
160
|
+
body = io ? nil : String.new(encoding: Encoding::BINARY)
|
|
161
|
+
decoder = nil
|
|
162
|
+
total = 0
|
|
163
|
+
content_length = nil
|
|
164
|
+
done = false
|
|
165
|
+
until done
|
|
166
|
+
type, flags, sid, payload = read_frame
|
|
167
|
+
case type
|
|
168
|
+
when TYPE_HEADERS, TYPE_CONTINUATION
|
|
169
|
+
next unless sid == stream_id || headers.nil?
|
|
170
|
+
block = type == TYPE_HEADERS ? strip_header_padding(payload, flags) : payload
|
|
171
|
+
header_blob << block
|
|
172
|
+
if flags & FLAG_END_HEADERS != 0
|
|
173
|
+
headers = @decoder.decode(header_blob)
|
|
174
|
+
if io
|
|
175
|
+
hdrs = CIHash.new
|
|
176
|
+
headers.each { |k, v| hdrs[k] = v unless k == ':status' }
|
|
177
|
+
decoder = stream_decoder(hdrs)
|
|
178
|
+
content_length = hdrs['content-length']&.to_i
|
|
179
|
+
end
|
|
180
|
+
end
|
|
181
|
+
done = true if sid == stream_id && flags & FLAG_END_STREAM != 0
|
|
182
|
+
when TYPE_DATA
|
|
183
|
+
if sid == stream_id
|
|
184
|
+
chunk = strip_data_padding(payload, flags)
|
|
185
|
+
update_recv_window(stream_id, payload.bytesize)
|
|
186
|
+
if io
|
|
187
|
+
piece = decoder == :raw ? chunk : decoder.inflate(chunk)
|
|
188
|
+
io.write(piece)
|
|
189
|
+
total += piece.bytesize
|
|
190
|
+
progress.call(total, content_length) if progress
|
|
191
|
+
else
|
|
192
|
+
body << chunk
|
|
193
|
+
end
|
|
194
|
+
end
|
|
195
|
+
done = true if sid == stream_id && flags & FLAG_END_STREAM != 0
|
|
196
|
+
else
|
|
197
|
+
dispatch_control_frame(type, flags, sid, payload)
|
|
198
|
+
done = true if type == TYPE_RST_STREAM && sid == stream_id
|
|
199
|
+
end
|
|
200
|
+
end
|
|
201
|
+
Http2Response.new(headers || [], body)
|
|
202
|
+
end
|
|
203
|
+
def dispatch_control_frame(type, flags, sid, payload)
|
|
204
|
+
case type
|
|
205
|
+
when TYPE_WINDOW_UPDATE
|
|
206
|
+
apply_window_update(sid, payload)
|
|
207
|
+
when TYPE_SETTINGS
|
|
208
|
+
if flags & FLAG_ACK == 0
|
|
209
|
+
apply_settings(payload)
|
|
210
|
+
write_frame(TYPE_SETTINGS, FLAG_ACK, 0, '')
|
|
211
|
+
end
|
|
212
|
+
when TYPE_PING
|
|
213
|
+
write_frame(TYPE_PING, FLAG_ACK, 0, payload) if flags & FLAG_ACK == 0
|
|
214
|
+
when TYPE_GOAWAY
|
|
215
|
+
@closed = true
|
|
216
|
+
when TYPE_RST_STREAM
|
|
217
|
+
raise Requests::ConnectionError, 'stream reset by server'
|
|
218
|
+
when TYPE_PUSH_PROMISE
|
|
219
|
+
psid = payload.byteslice(0, 4).unpack1('N') & 0x7fffffff
|
|
220
|
+
write_frame(TYPE_RST_STREAM, 0, psid, [0x8].pack('N'))
|
|
221
|
+
end
|
|
222
|
+
end
|
|
223
|
+
def apply_settings(payload)
|
|
224
|
+
i = 0
|
|
225
|
+
while i + 6 <= payload.bytesize
|
|
226
|
+
id = payload.byteslice(i, 2).unpack1('n')
|
|
227
|
+
val = payload.byteslice(i + 2, 4).unpack1('N')
|
|
228
|
+
@peer_initial_window = val if id == 4
|
|
229
|
+
@peer_max_frame_size = val if id == 5
|
|
230
|
+
i += 6
|
|
231
|
+
end
|
|
232
|
+
end
|
|
233
|
+
def apply_window_update(sid, payload)
|
|
234
|
+
inc = payload.unpack1('N') & 0x7fffffff
|
|
235
|
+
if sid == 0
|
|
236
|
+
@send_window += inc
|
|
237
|
+
else
|
|
238
|
+
(@streams[sid] ||= { send_window: @peer_initial_window, recv_window: DEFAULT_WINDOW })[:send_window] += inc
|
|
239
|
+
end
|
|
240
|
+
end
|
|
241
|
+
def update_recv_window(stream_id, n)
|
|
242
|
+
st = @streams[stream_id]
|
|
243
|
+
return unless st
|
|
244
|
+
st[:recv_window] -= n
|
|
245
|
+
@conn_recv_window -= n
|
|
246
|
+
if st[:recv_window] < 32_768
|
|
247
|
+
inc = DEFAULT_WINDOW - st[:recv_window]
|
|
248
|
+
write_frame(TYPE_WINDOW_UPDATE, 0, stream_id, [inc].pack('N'))
|
|
249
|
+
st[:recv_window] += inc
|
|
250
|
+
end
|
|
251
|
+
return unless @conn_recv_window < 32_768
|
|
252
|
+
inc = DEFAULT_WINDOW - @conn_recv_window
|
|
253
|
+
write_frame(TYPE_WINDOW_UPDATE, 0, 0, [inc].pack('N'))
|
|
254
|
+
@conn_recv_window += inc
|
|
255
|
+
end
|
|
256
|
+
def stream_decoder(hdrs)
|
|
257
|
+
case hdrs['content-encoding'].to_s.downcase
|
|
258
|
+
when 'gzip', 'x-gzip' then Zlib::Inflate.new(32 + Zlib::MAX_WBITS)
|
|
259
|
+
when 'deflate' then Zlib::Inflate.new
|
|
260
|
+
else :raw
|
|
261
|
+
end
|
|
262
|
+
end
|
|
263
|
+
def strip_header_padding(payload, flags)
|
|
264
|
+
pos = 0
|
|
265
|
+
padlen = 0
|
|
266
|
+
if flags & FLAG_PADDED != 0
|
|
267
|
+
padlen = payload.getbyte(0)
|
|
268
|
+
pos += 1
|
|
269
|
+
end
|
|
270
|
+
pos += 5 if flags & FLAG_PRIORITY != 0
|
|
271
|
+
payload.byteslice(pos, payload.bytesize - pos - padlen)
|
|
272
|
+
end
|
|
273
|
+
def strip_data_padding(payload, flags)
|
|
274
|
+
return payload if flags & FLAG_PADDED == 0
|
|
275
|
+
padlen = payload.getbyte(0)
|
|
276
|
+
payload.byteslice(1, payload.bytesize - 1 - padlen)
|
|
277
|
+
end
|
|
278
|
+
def read_frame
|
|
279
|
+
hdr = read_exactly(9)
|
|
280
|
+
length = (hdr.getbyte(0) << 16) | (hdr.getbyte(1) << 8) | hdr.getbyte(2)
|
|
281
|
+
type = hdr.getbyte(3)
|
|
282
|
+
flags = hdr.getbyte(4)
|
|
283
|
+
sid = hdr.byteslice(5, 4).unpack1('N') & 0x7fffffff
|
|
284
|
+
payload = length.positive? ? read_exactly(length) : ''
|
|
285
|
+
[type, flags, sid, payload]
|
|
286
|
+
end
|
|
287
|
+
def write_frame(type, flags, stream_id, payload)
|
|
288
|
+
len = payload.bytesize
|
|
289
|
+
hdr = [(len >> 16) & 0xff, (len >> 8) & 0xff, len & 0xff, type, flags].pack('C5') + [stream_id & 0x7fffffff].pack('N')
|
|
290
|
+
@socket.write(hdr + payload)
|
|
291
|
+
end
|
|
292
|
+
def read_exactly(n)
|
|
293
|
+
buf = String.new(encoding: Encoding::BINARY)
|
|
294
|
+
while buf.bytesize < n
|
|
295
|
+
chunk = @socket.read(n - buf.bytesize)
|
|
296
|
+
raise Requests::ConnectionError, 'connection closed by peer' if chunk.nil?
|
|
297
|
+
buf << chunk
|
|
298
|
+
end
|
|
299
|
+
buf
|
|
300
|
+
end
|
|
301
|
+
end
|
|
302
|
+
|
|
303
|
+
class Http2Adapter
|
|
304
|
+
attr_accessor :max_retries, :backoff_factor, :status_forcelist
|
|
305
|
+
def initialize(max_retries: 0, backoff_factor: 0, status_forcelist: [])
|
|
306
|
+
@max_retries = max_retries
|
|
307
|
+
@backoff_factor = backoff_factor
|
|
308
|
+
@status_forcelist = status_forcelist
|
|
309
|
+
@pool = {}
|
|
310
|
+
@alpn = {}
|
|
311
|
+
@fallback = HTTPAdapter.new(max_retries: max_retries, backoff_factor: backoff_factor, status_forcelist: status_forcelist)
|
|
312
|
+
end
|
|
313
|
+
def send_once(meth, url, hdrs, body, timeout, proxies, verify, cert, auth)
|
|
314
|
+
u = URI.parse(url)
|
|
315
|
+
sync_fallback!
|
|
316
|
+
return @fallback.send_once(meth, url, hdrs, body, timeout, proxies, verify, cert, auth) unless usable?(u, proxies)
|
|
317
|
+
conn = conn_for(u, verify, cert, timeout)
|
|
318
|
+
return @fallback.send_once(meth, url, hdrs, body, timeout, proxies, verify, cert, auth) if conn == :http1
|
|
319
|
+
attempt = 0
|
|
320
|
+
loop do
|
|
321
|
+
begin
|
|
322
|
+
auth.call(hdrs) if auth.respond_to?(:call) && !auth.is_a?(Requests::DigestAuth)
|
|
323
|
+
resp = perform(conn, meth, u, hdrs, body, timeout)
|
|
324
|
+
if @status_forcelist.include?(resp.code.to_i) && attempt < @max_retries
|
|
325
|
+
attempt += 1
|
|
326
|
+
sleep(retry_delay(resp, attempt))
|
|
327
|
+
conn = conn_for(u, verify, cert, timeout)
|
|
328
|
+
next
|
|
329
|
+
end
|
|
330
|
+
return resp
|
|
331
|
+
rescue Requests::ConnectionError, Requests::ConnectTimeout => e
|
|
332
|
+
drop_conn(u)
|
|
333
|
+
attempt += 1
|
|
334
|
+
raise e unless attempt <= @max_retries && idempotent?(meth)
|
|
335
|
+
sleep(@backoff_factor * attempt) if @backoff_factor.to_f > 0
|
|
336
|
+
conn = conn_for(u, verify, cert, timeout)
|
|
337
|
+
return @fallback.send_once(meth, url, hdrs, body, timeout, proxies, verify, cert, auth) if conn == :http1
|
|
338
|
+
end
|
|
339
|
+
end
|
|
340
|
+
end
|
|
341
|
+
def stream_to(meth, url, hdrs, body, timeout, proxies, verify, cert, auth, io, chunk_size: 65_536, progress: nil)
|
|
342
|
+
u = URI.parse(url)
|
|
343
|
+
sync_fallback!
|
|
344
|
+
unless usable?(u, proxies)
|
|
345
|
+
return @fallback.stream_to(meth, url, hdrs, body, timeout, proxies, verify, cert, auth, io, chunk_size: chunk_size, progress: progress)
|
|
346
|
+
end
|
|
347
|
+
conn = conn_for(u, verify, cert, timeout)
|
|
348
|
+
if conn == :http1
|
|
349
|
+
return @fallback.stream_to(meth, url, hdrs, body, timeout, proxies, verify, cert, auth, io, chunk_size: chunk_size, progress: progress)
|
|
350
|
+
end
|
|
351
|
+
auth.call(hdrs) if auth.respond_to?(:call) && !auth.is_a?(Requests::DigestAuth)
|
|
352
|
+
begin
|
|
353
|
+
header_list = build_headers(meth, u, hdrs)
|
|
354
|
+
_open_t, read_t = split_timeout(timeout)
|
|
355
|
+
with_timeout(read_t) { conn.request(meth, u, header_list, body, io: io, progress: progress) }
|
|
356
|
+
rescue Requests::ConnectionError, Requests::ConnectTimeout => e
|
|
357
|
+
drop_conn(u)
|
|
358
|
+
raise e
|
|
359
|
+
end
|
|
360
|
+
end
|
|
361
|
+
def decode_body(net_resp)
|
|
362
|
+
@fallback.decode_body(net_resp)
|
|
363
|
+
end
|
|
364
|
+
def close_pool
|
|
365
|
+
@pool.each_value { |c| c.close }
|
|
366
|
+
@pool.clear
|
|
367
|
+
@fallback.close_pool
|
|
368
|
+
end
|
|
369
|
+
private
|
|
370
|
+
def perform(conn, meth, u, hdrs, body, timeout)
|
|
371
|
+
header_list = build_headers(meth, u, hdrs)
|
|
372
|
+
_open_t, read_t = split_timeout(timeout)
|
|
373
|
+
with_timeout(read_t) { conn.request(meth, u, header_list, body) }
|
|
374
|
+
end
|
|
375
|
+
def with_timeout(read_t, &blk)
|
|
376
|
+
return blk.call unless read_t
|
|
377
|
+
::Timeout.timeout(read_t, Requests::ReadTimeout, &blk)
|
|
378
|
+
end
|
|
379
|
+
def sync_fallback!
|
|
380
|
+
@fallback.max_retries = @max_retries
|
|
381
|
+
@fallback.backoff_factor = @backoff_factor
|
|
382
|
+
@fallback.status_forcelist = @status_forcelist
|
|
383
|
+
end
|
|
384
|
+
def idempotent?(meth)
|
|
385
|
+
%w[GET HEAD OPTIONS PUT DELETE].include?(meth.to_s.upcase)
|
|
386
|
+
end
|
|
387
|
+
def retry_delay(resp, attempt)
|
|
388
|
+
ra = resp['retry-after']
|
|
389
|
+
if ra
|
|
390
|
+
return ra.to_i if ra =~ /\A\d+\z/
|
|
391
|
+
begin
|
|
392
|
+
d = Time.httpdate(ra) - Time.now
|
|
393
|
+
return d.positive? ? d : 0
|
|
394
|
+
rescue ArgumentError
|
|
395
|
+
nil
|
|
396
|
+
end
|
|
397
|
+
end
|
|
398
|
+
@backoff_factor.to_f > 0 ? @backoff_factor * attempt : 0
|
|
399
|
+
end
|
|
400
|
+
def usable?(u, proxies)
|
|
401
|
+
return false unless u.scheme == 'https'
|
|
402
|
+
return false if proxies && !proxies.empty? && (proxies[u.scheme] || proxies[u.scheme.to_sym])
|
|
403
|
+
true
|
|
404
|
+
end
|
|
405
|
+
def build_headers(meth, u, hdrs)
|
|
406
|
+
port = u.port && u.port != 443 ? ":#{u.port}" : ''
|
|
407
|
+
list = [[':method', meth.to_s.upcase], [':scheme', 'https'], [':path', u.request_uri], [':authority', "#{u.host}#{port}"]]
|
|
408
|
+
hdrs.each { |k, v| list << [k.to_s, v.to_s] unless k.to_s.downcase == 'host' }
|
|
409
|
+
list
|
|
410
|
+
end
|
|
411
|
+
def key_for(u)
|
|
412
|
+
"#{u.host}:#{u.port}"
|
|
413
|
+
end
|
|
414
|
+
def conn_for(u, verify, cert, timeout)
|
|
415
|
+
k = key_for(u)
|
|
416
|
+
cached = @pool[k]
|
|
417
|
+
return cached if cached && cached.alive?
|
|
418
|
+
@pool.delete(k)
|
|
419
|
+
return :http1 if @alpn[k] == 'http1'
|
|
420
|
+
sock = open_tls(u, verify, cert, timeout)
|
|
421
|
+
if sock.alpn_protocol == 'h2'
|
|
422
|
+
@alpn[k] = 'h2'
|
|
423
|
+
@pool[k] = Http2Connection.new(sock)
|
|
424
|
+
else
|
|
425
|
+
@alpn[k] = 'http1'
|
|
426
|
+
sock.close rescue nil
|
|
427
|
+
:http1
|
|
428
|
+
end
|
|
429
|
+
end
|
|
430
|
+
def drop_conn(u)
|
|
431
|
+
c = @pool.delete(key_for(u))
|
|
432
|
+
c.close if c
|
|
433
|
+
end
|
|
434
|
+
def open_tls(u, verify, cert, timeout)
|
|
435
|
+
open_t, _read_t = split_timeout(timeout)
|
|
436
|
+
tcp = open_t ? Socket.tcp(u.host, u.port, connect_timeout: open_t) : Socket.tcp(u.host, u.port)
|
|
437
|
+
ctx = OpenSSL::SSL::SSLContext.new
|
|
438
|
+
ctx.alpn_protocols = ['h2', 'http/1.1']
|
|
439
|
+
if verify == false
|
|
440
|
+
ctx.verify_mode = OpenSSL::SSL::VERIFY_NONE
|
|
441
|
+
else
|
|
442
|
+
ctx.verify_mode = OpenSSL::SSL::VERIFY_PEER
|
|
443
|
+
ctx.ca_file = verify.is_a?(String) ? verify : Requests::CA_FILE
|
|
444
|
+
end
|
|
445
|
+
if cert
|
|
446
|
+
ctx.cert = OpenSSL::X509::Certificate.new(File.read(cert[0]))
|
|
447
|
+
ctx.key = OpenSSL::PKey::RSA.new(File.read(cert[1]))
|
|
448
|
+
end
|
|
449
|
+
ssl = OpenSSL::SSL::SSLSocket.new(tcp, ctx)
|
|
450
|
+
ssl.hostname = u.host
|
|
451
|
+
ssl.sync_close = true
|
|
452
|
+
ssl.connect
|
|
453
|
+
ssl
|
|
454
|
+
rescue Errno::ETIMEDOUT, IO::TimeoutError
|
|
455
|
+
raise Requests::ConnectTimeout, "connect timeout: #{u}"
|
|
456
|
+
rescue OpenSSL::SSL::SSLError => e
|
|
457
|
+
raise Requests::SSLError, e.message
|
|
458
|
+
rescue SocketError, Errno::ECONNREFUSED, Errno::EHOSTUNREACH, Errno::ECONNRESET => e
|
|
459
|
+
raise Requests::ConnectionError, e.message
|
|
460
|
+
end
|
|
461
|
+
def split_timeout(t)
|
|
462
|
+
case t
|
|
463
|
+
when nil then [nil, nil]
|
|
464
|
+
when Array then [t[0], t[1]]
|
|
465
|
+
else [t, t]
|
|
466
|
+
end
|
|
467
|
+
end
|
|
468
|
+
end
|
|
469
|
+
end
|
data/lib/requests/sessions.rb
CHANGED
|
@@ -6,8 +6,8 @@ require 'time'
|
|
|
6
6
|
module Requests
|
|
7
7
|
class Session
|
|
8
8
|
attr_accessor :headers, :cookies, :auth, :params, :proxies, :verify,
|
|
9
|
-
:cert, :max_redirects, :timeout, :retries, :backoff_factor, :status_forcelist
|
|
10
|
-
def initialize
|
|
9
|
+
:cert, :max_redirects, :timeout, :retries, :backoff_factor, :status_forcelist, :trust_env
|
|
10
|
+
def initialize(http2: false)
|
|
11
11
|
@headers = Requests::Utils.default_headers
|
|
12
12
|
@cookies = Jar.new
|
|
13
13
|
@auth = nil
|
|
@@ -20,9 +20,11 @@ module Requests
|
|
|
20
20
|
@retries = 0
|
|
21
21
|
@backoff_factor = 0
|
|
22
22
|
@status_forcelist = []
|
|
23
|
+
@trust_env = true
|
|
23
24
|
@hooks = { response: [] }
|
|
24
|
-
@adapters = { 'https://' => HTTPAdapter.new, 'http://' => HTTPAdapter.new }
|
|
25
|
+
@adapters = { 'https://' => http2 ? Http2Adapter.new : HTTPAdapter.new, 'http://' => HTTPAdapter.new }
|
|
25
26
|
@default_adapters = @adapters.values.dup
|
|
27
|
+
@http2_adapter = http2 ? @adapters['https://'] : nil
|
|
26
28
|
end
|
|
27
29
|
def hooks
|
|
28
30
|
@hooks
|
|
@@ -31,6 +33,12 @@ module Requests
|
|
|
31
33
|
@adapters[prefix] = adapter
|
|
32
34
|
self
|
|
33
35
|
end
|
|
36
|
+
def http2!
|
|
37
|
+
@http2_adapter ||= Http2Adapter.new
|
|
38
|
+
@adapters['https://'] = @http2_adapter
|
|
39
|
+
@default_adapters = @adapters.values.dup
|
|
40
|
+
self
|
|
41
|
+
end
|
|
34
42
|
def get(url, **kw); request('GET', url, **kw); end
|
|
35
43
|
def post(url, **kw); request('POST', url, **kw); end
|
|
36
44
|
def put(url, **kw); request('PUT', url, **kw); end
|
|
@@ -46,12 +54,14 @@ module Requests
|
|
|
46
54
|
def put!(url, **kw); put(url, **kw).raise_for_status; end
|
|
47
55
|
def patch!(url, **kw); patch(url, **kw).raise_for_status; end
|
|
48
56
|
def delete!(url, **kw); delete(url, **kw).raise_for_status; end
|
|
57
|
+
def head!(url, **kw); head(url, **kw).raise_for_status; end
|
|
58
|
+
def options!(url, **kw); options(url, **kw).raise_for_status; end
|
|
49
59
|
def close
|
|
50
60
|
@adapters.values.uniq.each { |a| a.close_pool if a.respond_to?(:close_pool) }
|
|
51
61
|
true
|
|
52
62
|
end
|
|
53
63
|
def download(url, to:, chunk_size: 65536, resume: false, progress: nil, headers: nil, params: nil,
|
|
54
|
-
timeout: nil, proxies: nil, verify: nil, cert: nil, auth: nil)
|
|
64
|
+
timeout: nil, proxies: nil, verify: nil, cert: nil, auth: nil, http2: nil)
|
|
55
65
|
validate_url!(url)
|
|
56
66
|
full_url = build_url(url, merge_hash(@params, params))
|
|
57
67
|
hdrs = CIHash.new(@headers.to_h)
|
|
@@ -61,13 +71,13 @@ module Requests
|
|
|
61
71
|
hdrs['Range'] = "bytes=#{File.size(to)}-"
|
|
62
72
|
mode = 'ab'
|
|
63
73
|
end
|
|
64
|
-
adapter =
|
|
74
|
+
adapter = pick_adapter(full_url, http2)
|
|
65
75
|
adapter.max_retries = @retries
|
|
66
76
|
adapter.backoff_factor = @backoff_factor
|
|
67
77
|
use_auth = auth || @auth
|
|
68
78
|
use_auth.call(hdrs) if use_auth.respond_to?(:call) && !use_auth.is_a?(DigestAuth)
|
|
69
79
|
File.open(to, mode) do |f|
|
|
70
|
-
net_resp = adapter.stream_to('GET', full_url, hdrs, nil, timeout || @timeout,
|
|
80
|
+
net_resp = adapter.stream_to('GET', full_url, hdrs, nil, timeout || @timeout, effective_proxies(full_url, proxies),
|
|
71
81
|
verify.nil? ? @verify : verify, cert || @cert, nil, f,
|
|
72
82
|
chunk_size: chunk_size, progress: progress)
|
|
73
83
|
unless [200, 206].include?(net_resp.code.to_i)
|
|
@@ -78,7 +88,7 @@ module Requests
|
|
|
78
88
|
end
|
|
79
89
|
def request(method, url, params: nil, data: nil, json: nil, headers: nil, cookies: nil, files: nil,
|
|
80
90
|
auth: nil, timeout: nil, allow_redirects: true, proxies: nil, verify: nil, stream: false,
|
|
81
|
-
cert: nil, hooks: nil, retries: nil)
|
|
91
|
+
cert: nil, hooks: nil, retries: nil, http2: nil)
|
|
82
92
|
validate_url!(url)
|
|
83
93
|
use_auth = auth || @auth
|
|
84
94
|
merged_params = merge_hash(@params, params)
|
|
@@ -97,13 +107,14 @@ module Requests
|
|
|
97
107
|
ck = @cookies.to_header(cur_url)
|
|
98
108
|
ck = [ck, cookies.map { |k, v| "#{k}=#{v}" }.join('; ')].reject { |x| x.nil? || x.empty? }.join('; ') if cookies
|
|
99
109
|
hdrs['Cookie'] = ck unless ck.empty?
|
|
100
|
-
adapter =
|
|
110
|
+
adapter = pick_adapter(cur_url, http2)
|
|
101
111
|
adapter.max_retries = retries unless retries.nil?
|
|
102
112
|
adapter.max_retries = @retries if @retries.to_i > 0 && retries.nil? && default_adapter?(adapter)
|
|
103
113
|
adapter.backoff_factor = @backoff_factor if @backoff_factor.to_f > 0 && default_adapter?(adapter)
|
|
104
114
|
adapter.status_forcelist = @status_forcelist if !@status_forcelist.empty? && adapter.respond_to?(:status_forcelist=) && default_adapter?(adapter)
|
|
115
|
+
eff_proxies = effective_proxies(cur_url, proxies)
|
|
105
116
|
t0 = Time.now
|
|
106
|
-
net_resp = adapter.send_once(cur_method, cur_url, hdrs, body, timeout || @timeout,
|
|
117
|
+
net_resp = adapter.send_once(cur_method, cur_url, hdrs, body, timeout || @timeout, eff_proxies, verify.nil? ? @verify : verify,cert || @cert, use_auth)
|
|
107
118
|
elapsed = Time.now - t0
|
|
108
119
|
resp = to_response(adapter, net_resp, cur_url, elapsed, cur_method, hdrs, body)
|
|
109
120
|
@cookies.update(net_resp, cur_url)
|
|
@@ -111,7 +122,7 @@ module Requests
|
|
|
111
122
|
if use_auth.is_a?(DigestAuth) && resp.status_code == 401 && redirs.zero? && hist.empty?
|
|
112
123
|
use_auth.call(hdrs, meth: cur_method, url: cur_url, prev_resp: resp)
|
|
113
124
|
t1 = Time.now
|
|
114
|
-
net_resp = adapter.send_once(cur_method, cur_url, hdrs, body, timeout || @timeout,
|
|
125
|
+
net_resp = adapter.send_once(cur_method, cur_url, hdrs, body, timeout || @timeout, eff_proxies, verify.nil? ? @verify : verify,cert || @cert, nil)
|
|
115
126
|
resp = to_response(adapter, net_resp, cur_url, Time.now - t1, cur_method, hdrs, body)
|
|
116
127
|
@cookies.update(net_resp, cur_url)
|
|
117
128
|
resp.cookies = @cookies
|
|
@@ -149,6 +160,19 @@ module Requests
|
|
|
149
160
|
match = @adapters.keys.select { |prefix| url.start_with?(prefix) }.max_by(&:length)
|
|
150
161
|
match ? @adapters[match] : @adapters['https://']
|
|
151
162
|
end
|
|
163
|
+
def pick_adapter(url, http2)
|
|
164
|
+
return adapter_for(url) unless http2 == true
|
|
165
|
+
unless @http2_adapter
|
|
166
|
+
@http2_adapter = Http2Adapter.new
|
|
167
|
+
@default_adapters << @http2_adapter
|
|
168
|
+
end
|
|
169
|
+
@http2_adapter
|
|
170
|
+
end
|
|
171
|
+
def effective_proxies(url, proxies)
|
|
172
|
+
p = proxies || @proxies
|
|
173
|
+
return p if p && !p.empty?
|
|
174
|
+
@trust_env ? Requests::Utils.env_proxies(url) : (p || {})
|
|
175
|
+
end
|
|
152
176
|
def merge_hash(a, b)
|
|
153
177
|
return b unless a
|
|
154
178
|
return a unless b
|
data/lib/requests/utils.rb
CHANGED
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
+
require 'uri'
|
|
4
|
+
|
|
3
5
|
module Requests
|
|
4
6
|
module Utils
|
|
5
7
|
module_function
|
|
@@ -51,5 +53,21 @@ module Requests
|
|
|
51
53
|
def looks_like_html?(str)
|
|
52
54
|
!!(str.to_s.lstrip =~ /\A(<!DOCTYPE html|<html)/i)
|
|
53
55
|
end
|
|
56
|
+
def env_proxies(url)
|
|
57
|
+
host = URI.parse(url).host.to_s
|
|
58
|
+
no_proxy = ENV['NO_PROXY'] || ENV['no_proxy']
|
|
59
|
+
if no_proxy
|
|
60
|
+
list = no_proxy.split(',').map { |s| s.strip.downcase }.reject(&:empty?)
|
|
61
|
+
return {} if list.include?('*') || list.any? { |p| host.downcase == p || host.downcase.end_with?(".#{p.sub(/\A\./, '')}") }
|
|
62
|
+
end
|
|
63
|
+
out = {}
|
|
64
|
+
http = ENV['HTTP_PROXY'] || ENV['http_proxy']
|
|
65
|
+
https = ENV['HTTPS_PROXY'] || ENV['https_proxy']
|
|
66
|
+
out['http'] = http if http && !http.empty?
|
|
67
|
+
out['https'] = https if https && !https.empty?
|
|
68
|
+
out
|
|
69
|
+
rescue URI::InvalidURIError
|
|
70
|
+
{}
|
|
71
|
+
end
|
|
54
72
|
end
|
|
55
73
|
end
|
data/lib/requests/version.rb
CHANGED
data/lib/requests.rb
CHANGED
|
@@ -15,5 +15,7 @@ require_relative 'requests/auth'
|
|
|
15
15
|
require_relative 'requests/utils'
|
|
16
16
|
require_relative 'requests/models'
|
|
17
17
|
require_relative 'requests/adapters'
|
|
18
|
+
require_relative 'requests/hpack'
|
|
19
|
+
require_relative 'requests/http2'
|
|
18
20
|
require_relative 'requests/sessions'
|
|
19
21
|
require_relative 'requests/api'
|
metadata
CHANGED
|
@@ -1,13 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: requests_ruby
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.0.
|
|
4
|
+
version: 1.0.4
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- monji024
|
|
8
|
+
autorequire:
|
|
8
9
|
bindir: bin
|
|
9
10
|
cert_chain: []
|
|
10
|
-
date: 2026-
|
|
11
|
+
date: 2026-08-25 00:00:00.000000000 Z
|
|
11
12
|
dependencies:
|
|
12
13
|
- !ruby/object:Gem::Dependency
|
|
13
14
|
name: minitest
|
|
@@ -26,6 +27,7 @@ dependencies:
|
|
|
26
27
|
description: A stdlib-only Ruby HTTP client modeled after python's requests library.
|
|
27
28
|
Sessions, cookies, redirects, auth (Basic/Bearer/Digest), multipart uploads, streaming
|
|
28
29
|
responses, hooks, and a friendly exception hierarchy - no external gems required.
|
|
30
|
+
email:
|
|
29
31
|
executables: []
|
|
30
32
|
extensions: []
|
|
31
33
|
extra_rdoc_files: []
|
|
@@ -40,6 +42,8 @@ files:
|
|
|
40
42
|
- lib/requests/auth.rb
|
|
41
43
|
- lib/requests/cookies.rb
|
|
42
44
|
- lib/requests/exceptions.rb
|
|
45
|
+
- lib/requests/hpack.rb
|
|
46
|
+
- lib/requests/http2.rb
|
|
43
47
|
- lib/requests/models.rb
|
|
44
48
|
- lib/requests/sessions.rb
|
|
45
49
|
- lib/requests/status_codes.rb
|
|
@@ -56,6 +60,7 @@ metadata:
|
|
|
56
60
|
changelog_uri: https://github.com/monji024/requests_ruby/blob/main/CHANGELOG.md
|
|
57
61
|
bug_tracker_uri: https://github.com/monji024/requests_ruby/issues
|
|
58
62
|
documentation_uri: https://rubydoc.info/gems/requests_ruby
|
|
63
|
+
post_install_message:
|
|
59
64
|
rdoc_options: []
|
|
60
65
|
require_paths:
|
|
61
66
|
- lib
|
|
@@ -70,7 +75,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
70
75
|
- !ruby/object:Gem::Version
|
|
71
76
|
version: '0'
|
|
72
77
|
requirements: []
|
|
73
|
-
rubygems_version: 3.
|
|
78
|
+
rubygems_version: 3.3.15
|
|
79
|
+
signing_key:
|
|
74
80
|
specification_version: 4
|
|
75
81
|
summary: Requests is a simple, yet elegant, HTTP library.
|
|
76
82
|
test_files: []
|