fast_curl 0.3.1 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ebf05021f9b72fb51019dd2a89a0ca91ef6e0f7c060f1447f137f245a5aac52e
-  data.tar.gz: e0f3d17b12a4ce18ce28cb832d594d7716719edc9513213348228b06ce12104d
+  metadata.gz: 576cbc25d05c1a0857f8f762c3ed3744ad84c65457669c63bd5da62d904e72a0
+  data.tar.gz: b13fcb38e809a48ac3c85deb14a5ea56dbedb37803cad3cb3c2218a7bf88d483
 SHA512:
-  metadata.gz: 2629435cd5203198a39697a831f1ff2a3fe71c41798b98fbe40654ad53954b5fbcf010e2e2a9fdb209932f045c50684868ed5c5d66e9c30a5226dfe6ca71910e
-  data.tar.gz: 576179e43d4203cf2bda1e3cb0cd19d58f4a98ad724a7c5ab8e1eaf8554f9f1294eba99cbe0f399bc64ba41e4ce60cc981be6a2b1409dcc600b0699c38562355
+  metadata.gz: c3fed870f48939b5d016c9951d1c2503941496706af5fa0d157c964269a925190187e1b3378dcd8a43dea5c7a7d4c56bbf8edf0923f325fd74323b0166d97361
+  data.tar.gz: 6f13f0839354222ed5ff8ede5acf39137904226ddad7f860b8b08e3f2b3dcff6fda6b65d21ed8324673f4fda1cc24e8cf51b13a32a6681740bc012507e36072c

data/README.md

@@ -8,13 +8,14 @@ Ultra-fast parallel HTTP client for Ruby. C extension built on libcurl `curl_mul
 - **GVL release** — `rb_thread_call_without_gvl` during I/O, other Ruby threads keep running
 - **Fiber scheduler compatible** — works inside `Async do ... end` without blocking other fibers
 - **Three modes**: execute (all), first_execute (first N), stream_execute (yield as ready)
+- **Lazy Enumerable sources** — bounded request preparation for large or infinite streams
 - **Zero dependencies** — only libcurl (available everywhere)
 
 ## Installation
 
-**Requirements**: Ruby >= 3.1, libcurl
+**Requirements**: Ruby >= 2.7, libcurl
 
-> **Why Ruby 3.1?** The C extension uses `rb_fiber_scheduler_current`, `rb_fiber_scheduler_block` and `rb_fiber_scheduler_unblock` to properly yield control to the Fiber Scheduler during I/O. These APIs are stable starting from Ruby 3.1. Without them, there is no correct way for a C extension to cooperate with the scheduler — earlier approaches (`rb_thread_schedule`) hold the GVL and block other fibers.
+> **Fiber Scheduler support requires Ruby >= 3.1.** The C extension uses `rb_fiber_scheduler_current`, `rb_fiber_scheduler_block` and `rb_fiber_scheduler_unblock` to yield control to the Fiber Scheduler during I/O; these APIs are stable from Ruby 3.1. On 2.7 and 3.0 the extension builds and runs correctly, but that code is compiled out — so a request made inside a scheduler blocks the whole thread and **no sibling fiber runs until it finishes**. Other OS threads are unaffected, since the GVL is still released. If you use `async`, use Ruby >= 3.1.
 
 ```ruby
 gem 'fast_curl'
@@ -50,16 +51,65 @@ end
 
 ### POST with body and headers
 
+Be explicit about the encoding — `json:` and `form:` set the matching
+`Content-Type` for you:
+
 ```ruby
 FastCurl.post([
   {
     url: "https://api.example.com/users",
     headers: { "Authorization" => "Bearer token" },
-    body: { name: "John" }
+    json: { name: "John" }               # application/json
+  },
+  {
+    url: "https://api.example.com/login",
+    form: { user: "john", pass: "x" }    # application/x-www-form-urlencoded
+  },
+  {
+    url: "https://api.example.com/blob",
+    headers: { "Content-Type" => "application/xml" },
+    body: "<user/>"                      # sent as-is
   }
 ])
 ```
 
+A raw String `body:` without an explicit `Content-Type` is sent as
+`application/octet-stream`. A Hash `body:` is still encoded as JSON.
+
+Query parameters can be passed separately:
+
+```ruby
+FastCurl.get([{ url: "https://api.example.com/search", params: { q: "ruby", page: 2 } }])
+```
+
+### Lazy / bounded Enumerable sources
+
+`Array` keeps the existing fast path. Any other object responding to `#each` is
+consumed lazily, so large request sets do not need to be materialized first:
+
+```ruby
+requests = Enumerator.new do |y|
+  1_000_000.times do |i|
+    y << { url: "https://api.example.com/items/#{i}" }
+  end
+end
+
+FastCurl.stream_get(requests, connections: 20, buffer: 20) do |index, response|
+  puts "#{index}: #{response[:status]}"
+end
+```
+
+For lazy sources, at most `connections + buffer` requests are retained by
+`fast_curl`. The source itself may have produced one additional item before
+backpressure is applied, so a generator can observe a maximum look-ahead of
+`connections + buffer + 1`. `buffer` defaults to `connections`.
+
+`FastCurl.get` still returns all results in input order, so its result array is
+necessarily O(N). Use `stream_get` when the whole pipeline must stay bounded.
+Source exceptions and stream callback exceptions unwind the native multi loop
+and release active curl handles; an `ensure` in the source is also unwound on
+early completed `first_*` calls.
+
 ### First N responses (cancel the rest)
 
 ```ruby
@@ -70,6 +120,19 @@ result = FastCurl.first_get([
 ], count: 1)
 ```
 
+`accept:` can keep the race running until a response satisfies a predicate:
+
+```ruby
+result = FastCurl.first_get(
+  mirrors,
+  connections: 3,
+  accept: ->(response) { response[:status].between?(200, 299) }
+)
+```
+
+The predicate receives the response Hash. Rejected responses do not count
+toward `count`.
+
 ### Stream responses as they arrive
 
 ```ruby
@@ -78,13 +141,29 @@ FastCurl.stream_get(urls, connections: 50) do |index, response|
 end
 ```
 
-### Retry functionality (v0.2.0+)
+### Retries
+
+**Only idempotent methods (GET, HEAD, PUT, DELETE, OPTIONS) are retried.**
+Several retryable curl errors — `GOT_NOTHING`, `SEND_ERROR`, `RECV_ERROR`,
+`PARTIAL_FILE` — can occur *after* the server has already accepted and processed
+the request, so replaying a `POST` or `PATCH` may duplicate its side effects.
+If you know the endpoint is safe to replay (e.g. it takes an idempotency key),
+opt in with `retry_non_idempotent: true`.
+
+`timeout` applies to a single attempt. Use `total_timeout` to bound the whole
+call, including retries and backoff:
+
+```ruby
+FastCurl.get(urls, timeout: 5, retries: 3, total_timeout: 10_000)
+```
+
+Delays use exponential backoff with full jitter, starting from `retry_delay`.
 
 ```ruby
 # Automatic retry on network errors (timeout, connection issues)
 results = FastCurl.get([
   { url: "https://unreliable-api.com/data" }
-], retries: 3, retry_delay: 1000)  # 3 retries with 1s delay
+], retries: 3, retry_delay: 1000)  # base delay 1s, doubling with jitter
 
 # Retry on specific HTTP status codes
 results = FastCurl.get([
@@ -109,14 +188,35 @@ end
 
 ## Response format
 
+Every response — successful or not — has the same keys:
+
 ```ruby
 [index, {
-  status: 200,             # HTTP status code (0 on error)
-  headers: { "Key" => "Value" },
-  body: "response body"
+  status: 200,                    # HTTP status code, 0 on error
+  headers: { "content-type" => "application/json" },
+  body: "response body",
+  error: nil,                     # nil, or :curl_error / :invalid_request /
+                                  # :not_completed / :deadline_exceeded
+  error_code: nil,                # CURLcode when error == :curl_error
+  effective_url: "https://...",   # final URL after redirects
+  attempts: 1                     # attempts made, including the first
 }]
 ```
 
+Check `response[:error]` rather than `response[:status] == 200`; a status of `0`
+always means the request never produced an HTTP response.
+
+Header names are normalised to lower case (HTTP/2 sends them that way and
+HTTP/1.1 may not), and lookups are case-insensitive:
+
+```ruby
+response[:headers]["Content-Type"]   # => "application/json"
+response[:headers]["content-type"]   # => "application/json"
+```
+
+Repeated fields fold into one comma-separated String. `set-cookie` cannot be
+folded and is **always** an Array, even for a single cookie.
+
 ## Available methods
 
 | Method | Description |
@@ -137,10 +237,34 @@ end
 | Option | Default | Description |
 |---|---|---|
 | `connections` | 20 | Max parallel connections |
-| `timeout` | 30 | Per-request timeout in seconds |
-| `retries` | 1 | Number of retry attempts (0-10) |
-| `retry_delay` | 0 | Delay between retries in milliseconds |
+| `buffer` | `connections` | Lazy-source prefetch window; ignored for already-materialized Arrays |
+| `timeout` | 30 | Timeout for a single attempt, in seconds (1-300) |
+| `connect_timeout` | 10000 | Connection phase timeout, in milliseconds |
+| `total_timeout` | none | Wall-clock budget for the whole call, in milliseconds |
+| `retries` | 1 | Retry attempts for idempotent methods (0-10) |
+| `retry_delay` | 100 | Base backoff in milliseconds; doubles with jitter |
 | `retry_codes` | [] | HTTP status codes to retry on |
+| `retry_non_idempotent` | false | Also retry POST and PATCH |
+| `follow_redirects` | true | Follow `Location` headers |
+| `max_redirects` | 5 | Redirect limit (0-100) |
+| `accept` | none | `first_*` predicate receiving the response Hash |
+
+DNS results and TLS sessions are cached process-wide, so repeated calls to the
+same host skip resolution and can resume TLS. TCP connections are pooled only
+within a single call — see Known limitations.
+
+## Known limitations
+
+- TCP connections are not reused across separate calls; each call builds its own
+  `curl_multi` handle. Sharing libcurl's connection cache across concurrent
+  multi handles deadlocks or crashes, so only the DNS and TLS session caches are
+  shared.
+- The whole response body is buffered in memory (100 MB cap per response);
+  `stream_execute` streams *responses*, not bodies.
+- HTTP/2 multiplexing is enabled, but `connections` caps in-flight requests and
+  TCP connections with the same number, so multiplexing cannot be exploited
+  beyond that limit.
+- No multipart, cookie jar, proxy or auth helpers yet.
 
 ## Performance