requests_ruby 1.0.2 → 1.0.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 39c4cbcba9b1ee122bcd4ee5168716a9c7aed48b3ed463e1db32a3b2382789ad
-  data.tar.gz: b7fbe77ea7b887ba479fbf51aa186d350fa30f7b27135bb51a80d790ca0d335b
+  metadata.gz: 8b4146e6fbead1e412259712bcd0939b7971d542bd94a41007e210c9c51de37c
+  data.tar.gz: bbc39cd987b00d948be87f9b02251d2e058016123c6ad2fda16724d921575f24
 SHA512:
-  metadata.gz: 51632dd4038d0d55285a1b08b4ad2909855a29635237f087c10982a06d769dfecafefa899930e3a2c5fd329d498db41a2eb7e8a1ece0fe707f41f41c8d58b39b
-  data.tar.gz: d5bec1101e2e23e63d5a093b9fee6b2b044014970346cb0b70e7b52440d7e5f4dd20b2159e3082490648297cfd155e95c0e70de723e64284d0992439e2173969
+  metadata.gz: 658ca79dbadae76f08300daa6ef77b5302bf8c8b2a42877659b490ffdc8359cc07f4c61ff79b56e181f3a4173d9474f5447170420198ee3bb8ded744404f58d1
+  data.tar.gz: 37e529b01d2a6406e86332e1ce10835c1b124b7076ceba696da237c215a0e3dab48a570e526dd75b614a95bce426bd1ce87c2a76035867d78fd45f33fa2fef5e

data/CHANGELOG.md

@@ -1,6 +1,75 @@
 # Changelog
 
-## 2.1.0
+## 1.0.4
+
+Adds real HTTP/2 support and rounds off a handful of gaps that showed up
+against Python's `requests` (and `requests`+`urllib3`) feature set.
+
+### Added
+- **HTTP/2 support**, implemented from scratch on top of stdlib
+  `Socket`/`OpenSSL` (ALPN negotiation, HPACK header compression with
+  Huffman coding, frame multiplexing, flow control) - no external gems.
+  `Requests::Http2Adapter` is a drop-in `HTTPAdapter` replacement:
+  `Requests::Session.new(http2: true)`, `session.http2!`, or per-call
+  `http2: true` on any request/`download`. Transparently falls back to
+  HTTP/1.1 for plain `http://` URLs, proxied requests, or servers that
+  don't negotiate `h2` over ALPN.
+- `Session#trust_env` (default `true`) - proxies are now picked up from
+  `HTTP_PROXY`/`HTTPS_PROXY`/`NO_PROXY` (and lowercase variants) when no
+  explicit `proxies:` is set, matching python `requests`' default
+  behavior. Set `session.trust_env = false` to opt out.
+- Retry backoff now honors a server's `Retry-After` header (seconds or
+  HTTP-date) when retrying a status in `status_forcelist`, instead of
+  always using the fixed `backoff_factor * attempt` formula.
+- `Session#head!` / `#options!` and module-level `Requests.get!` /
+  `#post!` / `#put!` / `#patch!` / `#delete!` / `#head!` / `#options!` -
+  the raise-on-error helpers now cover every verb at both levels.
+- `Requests.session(**opts)` now forwards keyword args (e.g.
+  `Requests.session(http2: true)`) instead of always building a bare
+  `Session.new`.
+
+### Fixed
+- Resolved a leftover unresolved git merge conflict in this file.
+
+## 1.0.3
+
+Focused on the two things people actually hit walls on with a stdlib-only
+client: downloading big files without blowing up memory, and cookies that
+behave like cookies instead of a flat global hash.
+
+### Added
+- **Real connection pooling.** `HTTPAdapter` now keeps the underlying
+  `Net::HTTP` connection open and reuses it for subsequent requests to the
+  same host/port/proxy instead of doing a fresh TCP+TLS handshake every
+  single call. Connections are dropped and reopened automatically if the
+  server closes them or an error occurs. `Session#close` now actually
+  closes every pooled connection instead of being a no-op.
+- **`Session#download(url, to:, ...)` / `Requests.download(url, to:, ...)`
+  now streams the response straight to disk in chunks** instead of
+  buffering the whole file in memory first. Options: `chunk_size:`,
+  `resume: true` (sends a `Range` header and appends to a partially
+  downloaded file), `progress: ->(downloaded, total) { ... }`.
+  Handles gzip/deflate on the fly, one chunk at a time.
+- **`Requests::Jar` is now a real, domain-and-path-aware cookie jar**
+  instead of a flat name => value hash. Cookies set by one host are no
+  longer sent to every other host through the same session. Supports
+  `secure`, `http_only`, `expires`/`Max-Age`, explicit `set(name, value,
+  domain:, path:, secure:, http_only:, expires:)`, `for_domain(host)`,
+  and `for_url(url)`.
+- `Jar#save(path)` / `Requests::Jar.load(path)` - persist a cookie jar to
+  a JSON file and load it back later (survives process restarts, useful
+  for scripts that log in once and reuse the session cookies).
+- `HTTPAdapter#status_forcelist` - retry on specific response status codes
+  (e.g. `[502, 503, 504]`), on top of the existing connection-error retries.
+- `Session#put!` / `#patch!` / `#delete!` to match the existing `get!` /
+  `post!` raise-on-error helpers.
+
+### Fixed
+- A custom adapter passed to `Session#mount` with its own `max_retries`,
+  `backoff_factor`, or `status_forcelist` no longer gets silently
+  overwritten back to the session's defaults on the next request.
+
+## 1.0.2
 
 Big cleanup pass. Split the single `lib/requests.rb` file into a proper
 `lib/requests/*` structure, fixed several real bugs, and filled in a bunch

data/README.md

@@ -4,6 +4,7 @@ A zero-dependency, stdlib-only HTTP client for Ruby, modeled closely on Python's
 
 [![Gem Version](https://img.shields.io/badge/gem-requests__ruby-red?logo=rubygems)](https://rubygems.org/gems/requests_ruby)
 [![Ruby](https://img.shields.io/badge/ruby-%3E%3D%202.7-CC342D?logo=ruby)](https://www.ruby-lang.org)
+[![Gem Downloads](https://img.shields.io/gem/dt/requests_ruby)](https://rubygems.org/gems/requests_ruby)
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
 [![Zero Dependencies](https://img.shields.io/badge/dependencies-zero-brightgreen)](requests_ruby.gemspec)
 
@@ -27,6 +28,9 @@ A zero-dependency, stdlib-only HTTP client for Ruby, modeled closely on Python's
   - [Cookies](#cookies)
   - [Hooks](#hooks)
   - [Streaming & downloads](#streaming--downloads)
+  - [Downloading big files in chunks](#downloading-big-files-in-chunks)
+  - [Connection pooling](#connection-pooling)
+  - [HTTP/2](#http2)
   - [Custom transport adapters](#custom-transport-adapters)
   - [Status codes](#status-codes)
 - [API reference](#-api-reference)
@@ -219,13 +223,18 @@ Requests.get(url, timeout: [3, 10])    # [connect, read]
 # retry transient connection errors on idempotent methods (GET/HEAD/OPTIONS/PUT/DELETE)
 s = Requests::Session.new
 s.retries = 3
-s.backoff_factor = 0.5   # sleeps 0.5s, 1s, 1.5s between attempts
+s.backoff_factor = 0.5   # sleeps 0.5s, 1s, 1.5s between attempts, unless...
+s.status_forcelist = [429, 502, 503, 504] # ...also retry these response codes
 s.get(url)
 
 # or per-request
 Requests.get(url, retries: 3)
 ```
 
+A response in `status_forcelist` that comes back with a `Retry-After`
+header (seconds or an HTTP date) is retried after that delay instead of
+the `backoff_factor` formula.
+
 ### Proxies & SSL
 
 ```ruby
@@ -234,6 +243,12 @@ Requests.get(url, verify: false)               # skip cert verification (careful
 Requests.get(url, verify: '/path/to/ca.pem')   # custom CA bundle
 ```
 
+By default (`session.trust_env = true`), a `Session` without an explicit
+`proxies:` picks one up from the `HTTP_PROXY`/`HTTPS_PROXY`/`NO_PROXY`
+environment variables (lowercase names also work), same as python
+`requests`. Set `session.trust_env = false` to ignore the environment
+entirely.
+
 A bundled `cacert.pem` (Mozilla's CA bundle) ships with the gem and is used by default. Override it globally with the `REQUESTS_CA_FILE` environment variable.
 
 ### Cookies
@@ -247,6 +262,24 @@ s.cookies.set('theme', 'dark')
 s.get(url) # sent automatically on every request through this session
 ```
 
+`Requests::Jar` is a real cookie jar as of 1.0.3 — it tracks domain, path,
+`secure`, `http_only` and expiry per cookie, same as a browser would, so a
+cookie a session picked up from `api.example.com` won't leak into a request
+to some unrelated host through the same session:
+
+```ruby
+s = Requests::Session.new
+s.get('https://api.example.com/login') # server sets a cookie scoped to api.example.com
+s.get('https://other-service.example')  # that cookie is NOT sent here
+
+# set one explicitly with full control
+s.cookies.set('session_id', 'abc123', domain: 'api.example.com', path: '/', secure: true, http_only: true)
+
+# save/restore a session's cookies across process runs
+s.cookies.save('cookies.json')
+s.cookies = Requests::Jar.load('cookies.json')
+```
+
 ### Hooks
 
 ```ruby
@@ -266,7 +299,73 @@ r.iter_lines { |line| ... }
 Requests.download('https://example.com/file.zip', to: 'file.zip')
 ```
 
-> ℹ️ Note: unlike Python's `requests`, the body is fully read before `iter_content`/`iter_lines` yield anything — this library deliberately stays stdlib-only, and true lazy streaming would need extra machinery to keep a `net/http` connection open across method calls. See [Known limitations](#-known-limitations).
+> ℹ️ Note: `iter_content`/`iter_lines` on a regular `Response` still read the whole body first — this library stays stdlib-only, and true lazy streaming for arbitrary responses would need extra machinery to keep a `net/http` connection open across method calls. See [Known limitations](#-known-limitations). For actual large files, use `download` below — that one *is* real chunk-by-chunk streaming straight to disk.
+
+### Downloading big files in chunks
+
+As of 1.0.3, `download` doesn't buffer the file in memory at all — it reads
+from the socket in chunks and writes each one straight to disk, decoding
+gzip/deflate on the fly. This is the one to reach for instead of
+`Requests.get(url).save_to(path)` once files get past a few MB.
+
+```ruby
+Requests.download('https://example.com/big-file.zip', to: 'big-file.zip')
+
+# custom chunk size + a progress callback
+Requests.download('https://example.com/big-file.zip', to: 'big-file.zip',
+  chunk_size: 65536,
+  progress: ->(downloaded, total) { print "\r#{downloaded}/#{total}" })
+
+# resume an interrupted download (sends a Range header, appends to the file)
+Requests.download('https://example.com/big-file.zip', to: 'big-file.zip', resume: true)
+
+# same thing on a Session, so it reuses the pooled connection and session auth/headers
+s = Requests::Session.new
+s.headers['Authorization'] = 'Bearer xyz'
+s.download('https://example.com/private-file.zip', to: 'private-file.zip')
+```
+
+### Connection pooling
+
+As of 1.0.3, `HTTPAdapter` keeps the underlying `Net::HTTP` connection open
+and reuses it for further requests to the same host/port/proxy instead of
+paying for a new TCP+TLS handshake on every single call — this is the same
+idea as `urllib3`'s connection pool under python's `requests`. It's
+automatic and requires nothing from you:
+
+```ruby
+s = Requests::Session.new
+s.get('https://api.example.com/a') # opens + keeps the connection
+s.get('https://api.example.com/b') # reuses it, no new handshake
+s.close                            # closes every pooled connection when you're done
+```
+
+If a pooled connection goes stale (the server closed it, a timeout, whatever)
+it's transparently dropped and reopened on the next request; combined with
+`retries`/`backoff_factor` this makes long-lived sessions much more resilient
+than opening a fresh connection per call.
+
+### HTTP/2
+
+As of 1.0.4, `Requests::Http2Adapter` speaks HTTP/2 directly over
+`Socket`/`OpenSSL` - ALPN negotiation, HPACK header compression, and
+stream multiplexing/flow control, all stdlib, no external gem:
+
+```ruby
+s = Requests::Session.new(http2: true)
+s.get('https://api.example.com/a') # negotiates h2 over ALPN, reuses the connection
+s.get('https://api.example.com/b') # same connection, new stream
+
+# or opt in per call on a plain session
+Requests.get('https://api.example.com', http2: true)
+resp = Requests::Session.new.tap(&:http2!).get('https://api.example.com')
+```
+
+If the server doesn't advertise `h2` during the TLS handshake, or the URL
+is plain `http://`, or a proxy is configured, requests on that host
+transparently fall back to the regular `HTTPAdapter` (HTTP/1.1) - you get
+the same `Response` object either way and don't need to know which
+transport actually served the request.
 
 ### Custom transport adapters
 
@@ -298,12 +397,58 @@ Requests.codes.im_a_teapot # 418, yes really
 |---|---|---|
 | `Requests.get/post/put/patch/delete/head/options` | `session.get/post/put/patch/delete/head/options` | `status_code`, `status` |
 | `Requests.request(method, url, **opts)` | `session.request(method, url, **opts)` | `ok?`, `redirect?` |
-| `Requests.session` → new `Session` | `session.get!/post!` (raise on error) | `text`, `content`, `json` |
-| `Requests.download(url, to:)` | `session.mount(prefix, adapter)` | `headers`, `cookies` |
-| `Requests.codes` | `session.hooks[:response]` | `elapsed`, `history`, `url` |
-|  | `session.headers`, `.cookies`, `.auth`, `.proxies`, `.retries` | `raise_for_status`, `save_to`, `links` |
-
-Every request-level method accepts: `params`, `data`, `json`, `headers`, `cookies`, `files`, `auth`, `timeout`, `allow_redirects`, `proxies`, `verify`, `cert`, `hooks`, `retries`.
+| `Requests.get!/post!/put!/patch!/delete!/head!/options!` (raise on error) | `session.get!/post!/put!/patch!/delete!/head!/options!` (raise on error) | `text`, `content`, `json` |
+| `Requests.session(**opts)` → new `Session` | `session.mount(prefix, adapter)`, `session.close` | `headers`, `cookies` |
+| `Requests.download(url, to:, chunk_size:, resume:, progress:)` | `session.download(url, to:, ...)` (same, but reuses the session's pool/auth/headers) | `elapsed`, `history`, `url` |
+| `Requests.codes` | `session.http2!`, `Session.new(http2: true)`, or `http2: true` per request | `raise_for_status`, `save_to`, `links` |
+|  | `session.hooks[:response]` |  |
+|  | `session.headers`, `.cookies`, `.auth`, `.proxies`, `.trust_env`, `.retries`, `.backoff_factor`, `.status_forcelist` |  |
+|  | `Requests::Jar#save(path)` / `Requests::Jar.load(path)` |  |
+|  | `HTTPAdapter.new(max_retries:, backoff_factor:, status_forcelist:)`, `Http2Adapter.new(...)` |  |
+
+Every request-level method accepts: `params`, `data`, `json`, `headers`, `cookies`, `files`, `auth`, `timeout`, `allow_redirects`, `proxies`, `verify`, `cert`, `hooks`, `retries`, `http2`.
+
+## Python parity cheat sheet
+
+| Python `requests` | `requests_ruby` |
+|---|---|
+| `requests.get(url, params={...})` | `Requests.get(url, params: {...})` |
+| `requests.post(url, json={...})` | `Requests.post(url, json: {...})` |
+| `r.status_code` | `r.status_code` |
+| `r.ok` | `r.ok?` |
+| `r.text` / `r.content` | `r.text` / `r.content` |
+| `r.json()` | `r.json` |
+| `r.raise_for_status()` | `r.raise_for_status` |
+| `r.headers['x']` | `r.headers['x']` (case-insensitive both ways) |
+| `s = requests.Session()` | `s = Requests::Session.new` |
+| `s.mount('https://', adapter)` | `s.mount('https://', adapter)` |
+| `requests.auth.HTTPBasicAuth(u, p)` | `Requests::BasicAuth.new(u, p)` |
+| `requests.exceptions.Timeout` | `Requests::Timeoutable` (module, catches both) |
+| `r.iter_content(chunk_size=...)` | `r.iter_content(chunk_size: ...)` |
+| `s.cookies.get_dict()` | `s.cookies.to_h` |
+| n/a | `Requests.download(url, to:, resume:, progress:)` — no python equivalent, this one's ours |
+
+## Known limitations
+
+- `iter_content`/`iter_lines` on a regular response read the whole body up
+  front rather than lazily streaming it off the socket — `download` (added
+  in 1.0.3) is the one that streams for real, use that for big files.
+- HTTP/2 (1.0.4) only applies to direct `https://` connections - there's
+  no `h2c` (cleartext HTTP/2) support, and requests through a proxy always
+  use HTTP/1.1. No automatic NTLM/Kerberos/OAuth either — bring your own
+  `auth:` object (anything with `#call(headers)` works) if you need those.
+- The HTTP/2 adapter applies `timeout:` as one coarse deadline around the
+  whole request/response exchange rather than resetting it on every
+  individual socket read the way the HTTP/1.1 adapter's `Net::HTTP`
+  read-timeout does.
+- `verify: '/path/to/ca.pem'` accepts a single bundle file, not a
+  directory of certs (`Net::HTTP` itself only takes `ca_file` or
+  `ca_path`, and this gem only wires up the former today).
+- The connection pool added in 1.0.3 (HTTP/1.1 and HTTP/2 alike) is per-
+  `Session`/per-adapter instance and isn't thread-safe for concurrent
+  requests on the *same* `Session` object — use one `Session` per thread,
+  or one connection per thread, same advice as most connection-pooling
+  HTTP clients.
 
 
 ## 🔧 Development
@@ -326,6 +471,10 @@ Bug reports and pull requests are welcome. A few ground rules:
 2. Add a test in `spec/requests_spec.rb` for anything you fix or add.
 3. Keep the Python-parity naming where it makes sense, but don't force it where it doesn't fit Ruby idiom (`?`-suffixed predicates, etc.).
 
+## Support the project
+
+If requests_ruby helped you, consider starring the repository ⭐
+
 ## License
 
 [MIT](LICENSE)

data/lib/requests/adapters.rb

@@ -5,6 +5,7 @@ require 'openssl'
 require 'uri'
 require 'zlib'
 require 'stringio'
+require 'time'
 
 module Requests
   class HTTPAdapter
@@ -16,24 +17,72 @@ module Requests
       'DELETE' => Net::HTTP::Delete,
       'HEAD' => Net::HTTP::Head,
       'OPTIONS' => Net::HTTP::Options}.freeze
-    attr_accessor :max_retries, :backoff_factor
-    def initialize(max_retries: 0, backoff_factor: 0)
+    attr_accessor :max_retries, :backoff_factor, :status_forcelist
+    def initialize(max_retries: 0, backoff_factor: 0, status_forcelist: [])
       @max_retries = max_retries
       @backoff_factor = backoff_factor
+      @status_forcelist = status_forcelist
+      @pool = {}
     end
     def send_once(meth, url, hdrs, body, timeout, proxies, verify, cert, auth)
       attempt = 0
-      begin
-        do_send(meth, url, hdrs, body, timeout, proxies, verify, cert, auth)
-      rescue Requests::ConnectionError, Requests::ConnectTimeout => e
-        attempt += 1
-        if attempt <= @max_retries && idempotent?(meth)
+      loop do
+        begin
+          r = do_send(meth, url, hdrs, body, timeout, proxies, verify, cert, auth)
+          if @status_forcelist.include?(r.code.to_i) && attempt < @max_retries
+            attempt += 1
+            sleep(retry_delay(r, attempt))
+            next
+          end
+          return r
+        rescue Requests::ConnectionError, Requests::ConnectTimeout => e
+          attempt += 1
+          raise e unless attempt <= @max_retries && idempotent?(meth)
           sleep(@backoff_factor * attempt) if @backoff_factor.to_f > 0
-          retry
         end
-        raise e
       end
     end
+    def close_pool
+      @pool.each_value { |h| (h.finish if h.started?) rescue nil }
+      @pool.clear
+    end
+    def stream_to(meth, url, hdrs, body, timeout, proxies, verify, cert, auth, io, chunk_size: 65536, progress: nil)
+      u = URI.parse(url)
+      auth.call(hdrs) if auth.respond_to?(:call) && !auth.is_a?(Requests::DigestAuth)
+      open_t, read_t = split_timeout(timeout)
+      http = conn_for(u, proxies, verify, cert, open_t, read_t)
+      klass = METHS[meth] || Net::HTTP::Get
+      req = klass.new(u.request_uri)
+      hdrs.each { |k, v| req[k] = v }
+      req.body = body if body
+      net_resp = nil
+      total = 0
+      begin
+        http.request(req) do |nr|
+          net_resp = nr
+          dec = stream_decoder(nr)
+          nr.read_body do |chunk|
+            piece = dec == :raw ? chunk : dec.inflate(chunk)
+            io.write(piece)
+            total += piece.bytesize
+            progress.call(total, nr.content_length) if progress
+          end
+        end
+      rescue Net::OpenTimeout
+        drop_conn(u, proxies)
+        raise Requests::ConnectTimeout, "connect timeout: #{url}"
+      rescue Net::ReadTimeout
+        drop_conn(u, proxies)
+        raise Requests::ReadTimeout, "read timeout: #{url}"
+      rescue OpenSSL::SSL::SSLError => e
+        drop_conn(u, proxies)
+        raise Requests::SSLError, e.message
+      rescue EOFError, IOError, Errno::ECONNRESET, Errno::EPIPE, SocketError, Errno::ECONNREFUSED, Errno::EHOSTUNREACH, Errno::ETIMEDOUT => e
+        drop_conn(u, proxies)
+        raise Requests::ConnectionError, e.message
+      end
+      net_resp
+    end
     def decode_body(net_resp)
       raw = net_resp.body
       return '' if raw.nil?
@@ -56,33 +105,78 @@ module Requests
     def idempotent?(meth)
       %w[GET HEAD OPTIONS PUT DELETE].include?(meth.to_s.upcase)
     end
+    def retry_delay(resp, attempt)
+      ra = resp['retry-after']
+      if ra
+        return ra.to_i if ra =~ /\A\d+\z/
+        begin
+          d = Time.httpdate(ra) - Time.now
+          return d.positive? ? d : 0
+        rescue ArgumentError
+          nil
+        end
+      end
+      @backoff_factor.to_f > 0 ? @backoff_factor * attempt : 0
+    end
     def do_send(meth, url, hdrs, body, timeout, proxies, verify, cert, auth)
       u = URI.parse(url)
       auth.call(hdrs) if auth.respond_to?(:call) && !auth.is_a?(Requests::DigestAuth)
-      popt = proxy_opts(u, proxies)
       open_t, read_t = split_timeout(timeout)
-      http = Net::HTTP.new(u.host, u.port, *popt)
-      configure_ssl(http, u, verify, cert)
-      http.open_timeout = open_t if open_t
-      http.read_timeout = read_t if read_t
+      http = conn_for(u, proxies, verify, cert, open_t, read_t)
       begin
-        http.start do |h|
-          klass = METHS[meth] || Net::HTTP::Get
-          req = klass.new(u.request_uri)
-          hdrs.each { |k, v| req[k] = v }
-          req.body = body if body
-          h.request(req)
-        end
+        klass = METHS[meth] || Net::HTTP::Get
+        req = klass.new(u.request_uri)
+        hdrs.each { |k, v| req[k] = v }
+        req.body = body if body
+        http.request(req)
       rescue Net::OpenTimeout
+        drop_conn(u, proxies)
         raise Requests::ConnectTimeout, "connect timeout: #{url}"
       rescue Net::ReadTimeout
+        drop_conn(u, proxies)
         raise Requests::ReadTimeout, "read timeout: #{url}"
       rescue OpenSSL::SSL::SSLError => e
+        drop_conn(u, proxies)
         raise Requests::SSLError, e.message
-      rescue SocketError, Errno::ECONNREFUSED, Errno::EHOSTUNREACH, Errno::ETIMEDOUT, Errno::ECONNRESET => e
+      rescue EOFError, IOError, Errno::ECONNRESET, Errno::EPIPE, SocketError, Errno::ECONNREFUSED, Errno::EHOSTUNREACH, Errno::ETIMEDOUT => e
+        drop_conn(u, proxies)
         raise Requests::ConnectionError, e.message
       end
     end
+    def conn_for(u, proxies, verify, cert, ot, rt)
+      key = pool_key(u, proxies)
+      h = @pool[key]
+      if h && h.started?
+        h.open_timeout = ot if ot
+        h.read_timeout = rt if rt
+        return h
+      end
+      popt = proxy_opts(u, proxies)
+      h = Net::HTTP.new(u.host, u.port, *popt)
+      configure_ssl(h, u, verify, cert)
+      h.open_timeout = ot if ot
+      h.read_timeout = rt if rt
+      h.keep_alive_timeout = 30
+      h.start
+      @pool[key] = h
+      h
+    end
+    def drop_conn(u, proxies)
+      key = pool_key(u, proxies)
+      h = @pool.delete(key)
+      (h.finish if h && h.started?) rescue nil
+    end
+    def pool_key(u, proxies)
+      p = proxy_opts(u, proxies)
+      "#{u.scheme}|#{u.host}|#{u.port}|#{p.join(',')}"
+    end
+    def stream_decoder(nr)
+      case nr['content-encoding'].to_s.downcase
+      when 'gzip', 'x-gzip' then Zlib::Inflate.new(32 + Zlib::MAX_WBITS)
+      when 'deflate' then Zlib::Inflate.new
+      else :raw
+      end
+    end
     def configure_ssl(http, uri, verify, cert)
       return unless uri.scheme == 'https'
       http.use_ssl = true

data/lib/requests/api.rb

@@ -14,12 +14,17 @@ module Requests
     kw[:allow_redirects] = kw.fetch(:allow_redirects, false)
     request('HEAD', url, **kw)
   end
-  def self.session
-    Session.new
+  def self.get!(url, **kw); get(url, **kw).raise_for_status; end
+  def self.post!(url, **kw); post(url, **kw).raise_for_status; end
+  def self.put!(url, **kw); put(url, **kw).raise_for_status; end
+  def self.patch!(url, **kw); patch(url, **kw).raise_for_status; end
+  def self.delete!(url, **kw); delete(url, **kw).raise_for_status; end
+  def self.head!(url, **kw); head(url, **kw).raise_for_status; end
+  def self.options!(url, **kw); options(url, **kw).raise_for_status; end
+  def self.session(**kw)
+    Session.new(**kw)
   end
   def self.download(url, to:, **kw)
-    resp = get(url, **kw)
-    resp.raise_for_status
-    resp.save_to(to)
+    Session.new.download(url, to: to, **kw)
   end
 end

data/lib/requests/cookies.rb

@@ -1,55 +1,128 @@
 # frozen_string_literal: true
 
+require 'time'
+require 'json'
+
 module Requests
   class Jar
     include Enumerable
+    Ck = Struct.new(:name, :value, :domain, :path, :secure, :http_only, :expires)
+
     def initialize
-      @c = {}
+      @s = {}
     end
     def [](n)
-      @c[n]
+      e = @s.values.reverse.find { |c| c.name == n }
+      e && e.value
     end
     def []=(n, v)
-      @c[n] = v
+      set(n, v)
     end
     def get(name, default = nil)
-      @c.fetch(name, default)
+      v = self[name]
+      v.nil? ? default : v
     end
-    def set(name, value)
-      @c[name] = value
+    def set(name, value, domain: '', path: '/', secure: false, http_only: false, expires: nil)
+      @s[ck(domain, path, name)] = Ck.new(name, value, domain, path, secure, http_only, expires)
       self
     end
+    alias_method :set_cookie, :set
     def delete(name)
-      @c.delete(name)
+      @s.delete_if { |_, c| c.name == name }
     end
     def clear
-      @c.clear
+      @s.clear
     end
     def to_h
-      @c.dup
+      h = {}
+      @s.each_value { |c| h[c.name] = c.value }
+      h
     end
     def to_a
-      @c.to_a
+      to_h.to_a
     end
     def each(&b)
-      @c.each(&b)
+      return enum_for(:each) unless block_given?
+      @s.each_value { |c| yield c.name, c.value }
+    end
+    def all
+      @s.values
     end
     def empty?
-      @c.empty?
+      @s.empty?
+    end
+    def for_url(url)
+      u = safe_uri(url)
+      now = Time.now
+      @s.values.select do |c|
+        next false if c.expires && c.expires < now
+        next false if c.secure && u.scheme != 'https'
+        h = u.host.to_s
+        dm = c.domain.to_s.empty? || h == c.domain || h.end_with?(".#{c.domain}")
+        pp = u.path.to_s
+        pm = pp.empty? ? c.path == '/' : pp.start_with?(c.path)
+        dm && pm
+      end
+    end
+    def for_domain(host)
+      @s.values.select { |c| c.domain.to_s.empty? || host.to_s == c.domain || host.to_s.end_with?(".#{c.domain}") }
     end
-    def to_header
-      @c.map { |k, v| "#{k}=#{v}" }.join('; ')
+    def to_header(url = nil)
+      list = url ? for_url(url) : @s.values.select { |c| !(c.expires && c.expires < Time.now) }
+      list.map { |c| "#{c.name}=#{c.value}" }.join('; ')
     end
-    def update(net_resp)
-      lines = net_resp.get_fields('set-cookie') || []
-      lines.each do |l|
-        kv = l.split(';').first
+    def update(net_resp, url = nil)
+      u = url ? safe_uri(url) : nil
+      (net_resp.get_fields('set-cookie') || []).each do |line|
+        parts = line.split(';').map(&:strip)
+        kv = parts.shift
         next unless kv
         k, v = kv.split('=', 2)
         next unless k
-        @c[k.strip] = v.to_s.strip
+        at = {}
+        parts.each do |p|
+          pk, pv = p.split('=', 2)
+          at[pk.to_s.downcase] = pv
+        end
+        dom = at['domain'].to_s.sub(/\A\./, '')
+        dom = u.host.to_s if dom.empty? && u
+        exp = nil
+        if at['max-age']
+          exp = (Time.now + at['max-age'].to_i rescue nil)
+        elsif at['expires']
+          exp = (Time.parse(at['expires']) rescue nil)
+        end
+        set(k, v.to_s, domain: dom, path: at['path'] || '/', secure: at.key?('secure'),
+            http_only: at.key?('httponly'), expires: exp)
       end
     end
+    def save(path)
+      data = @s.values.map do |c|
+        { name: c.name, value: c.value, domain: c.domain, path: c.path, secure: c.secure,
+          http_only: c.http_only, expires: c.expires && c.expires.iso8601 }
+      end
+      File.write(path, JSON.generate(data))
+      path
+    end
+    def self.load(path)
+      j = new
+      arr = JSON.parse(File.read(path))
+      arr.each do |c|
+        j.set(c['name'], c['value'], domain: c['domain'].to_s, path: c['path'] || '/',
+              secure: !!c['secure'], http_only: !!c['http_only'],
+              expires: (c['expires'] ? Time.parse(c['expires']) : nil))
+      end
+      j
+    end
+    private
+    def ck(domain, path, name)
+      "#{domain}|#{path}|#{name}"
+    end
+    def safe_uri(url)
+      URI.parse(url)
+    rescue
+      URI.parse('')
+    end
   end
   CookieJar = Jar
 end