fast_curl 0.4.0 → 0.5.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 32aca3a9aec734c69bc40aa9b84b519aeddc15defc3ad3a9e8f278f63d9f0ffb
-  data.tar.gz: 8037377b3d3e5ab02f4171c3f4ae56e6f42552a6cb5c6d7e2f5fd54921e2cc71
+  metadata.gz: e0c5dac84b62da99fa420dc28437ca3284ddd4f73fd970c7730df3c8a2a7ba5b
+  data.tar.gz: 7455385e6a1b1c76467e67909bb252a70a6eac465846aacdf988d9e934fb7d79
 SHA512:
-  metadata.gz: 670eefe8b9fa597e947ae0858bda833d35d15914fbd62e3093483cdf8c2c68560c121586c4b01eec75f81d3a4f6d49d04829d5711180178247f051cb510c8cd6
-  data.tar.gz: 709792e880465cc09de983246936ce66a843f5b7899e69ac6f5a390cef35b4cdc8baaf1dbda36abe39627cc2fe457e994076329ab746887d000d1a089e649af7
+  metadata.gz: 5f5b3c2c3797436d23ec37e9f2997c340a5e50b05a4ef9dc82a699093b01fc7a3097c720e14adba1fe6ad3e132e29d90893e01c4bd8a81fbdde3043b9067a727
+  data.tar.gz: 5a990531123a9bd72ccc082a5076a5dd1f6eed90f455e92c77c336be33ed964f3ad9a6cef3d59e6322cf378223399505edcaa6d27dd7a0619d71f5c600870bba

data/README.md

@@ -8,6 +8,7 @@ Ultra-fast parallel HTTP client for Ruby. C extension built on libcurl `curl_mul
 - **GVL release** — `rb_thread_call_without_gvl` during I/O, other Ruby threads keep running
 - **Fiber scheduler compatible** — works inside `Async do ... end` without blocking other fibers
 - **Three modes**: execute (all), first_execute (first N), stream_execute (yield as ready)
+- **Lazy Enumerable sources** — bounded request preparation for large or infinite streams
 - **Zero dependencies** — only libcurl (available everywhere)
 
 ## Installation
@@ -81,6 +82,34 @@ Query parameters can be passed separately:
 FastCurl.get([{ url: "https://api.example.com/search", params: { q: "ruby", page: 2 } }])
 ```
 
+### Lazy / bounded Enumerable sources
+
+`Array` keeps the existing fast path. Any other object responding to `#each` is
+consumed lazily, so large request sets do not need to be materialized first:
+
+```ruby
+requests = Enumerator.new do |y|
+  1_000_000.times do |i|
+    y << { url: "https://api.example.com/items/#{i}" }
+  end
+end
+
+FastCurl.stream_get(requests, connections: 20, buffer: 20) do |index, response|
+  puts "#{index}: #{response[:status]}"
+end
+```
+
+For lazy sources, at most `connections + buffer` requests are retained by
+`fast_curl`. The source itself may have produced one additional item before
+backpressure is applied, so a generator can observe a maximum look-ahead of
+`connections + buffer + 1`. `buffer` defaults to `connections`.
+
+`FastCurl.get` still returns all results in input order, so its result array is
+necessarily O(N). Use `stream_get` when the whole pipeline must stay bounded.
+Source exceptions and stream callback exceptions unwind the native multi loop
+and release active curl handles; an `ensure` in the source is also unwound on
+early completed `first_*` calls.
+
 ### First N responses (cancel the rest)
 
 ```ruby
@@ -91,6 +120,19 @@ result = FastCurl.first_get([
 ], count: 1)
 ```
 
+`accept:` can keep the race running until a response satisfies a predicate:
+
+```ruby
+result = FastCurl.first_get(
+  mirrors,
+  connections: 3,
+  accept: ->(response) { response[:status].between?(200, 299) }
+)
+```
+
+The predicate receives the response Hash. Rejected responses do not count
+toward `count`.
+
 ### Stream responses as they arrive
 
 ```ruby
@@ -195,6 +237,7 @@ folded and is **always** an Array, even for a single cookie.
 | Option | Default | Description |
 |---|---|---|
 | `connections` | 20 | Max parallel connections |
+| `buffer` | `connections` | Lazy-source prefetch window; ignored for already-materialized Arrays |
 | `timeout` | 30 | Timeout for a single attempt, in seconds (1-300) |
 | `connect_timeout` | 10000 | Connection phase timeout, in milliseconds |
 | `total_timeout` | none | Wall-clock budget for the whole call, in milliseconds |
@@ -204,6 +247,7 @@ folded and is **always** an Array, even for a single cookie.
 | `retry_non_idempotent` | false | Also retry POST and PATCH |
 | `follow_redirects` | true | Follow `Location` headers |
 | `max_redirects` | 5 | Redirect limit (0-100) |
+| `accept` | none | `first_*` predicate receiving the response Hash |
 
 DNS results and TLS sessions are cached process-wide, so repeated calls to the
 same host skip resolution and can resume TLS. TCP connections are pooled only
@@ -234,27 +278,27 @@ grows linearly while parallel clients stay near ~1s plus network overhead.
 
 | Scenario                        | Net::HTTP sequential | fast_curl (thread) | fast_curl (fiber/Async) | Async::HTTP::Internet |
 |---------------------------------|---------------------:|-------------------:|------------------------:|----------------------:|
-| 4 requests × 1s, conn=4         |                8.27s |              2.36s |                   2.13s |                 2.56s |
-| 10 requests × 1s, conn=10       |               20.92s |              3.49s |                   5.23s |                 3.83s |
-| 20 requests × 1s, conn=5        |               42.56s |              2.94s |                   2.90s |                12.14s |
-| 200 requests × 1s, conn=20      |                   —  |             22.19s |                  21.77s |                23.59s |
+| 4 requests × 1s, conn=4         |                5.82s |              1.68s |                   1.65s |                 1.93s |
+| 10 requests × 1s, conn=10       |               17.51s |              1.70s |                   1.68s |                 1.89s |
+| 20 requests × 1s, conn=5        |               34.66s |              5.69s |                   6.71s |                 5.79s |
+| 200 requests × 1s, conn=20      |                   —  |             14.29s |                  14.29s |                 7.27s |
 
 ### Speedup vs Net::HTTP (median)
 
 | Scenario                          | fast_curl (thread) | fast_curl (fiber) | Async::HTTP |
 |-----------------------------------|-------------------:|------------------:|------------:|
-| 4 requests × 1s                   |           **3.5x** |              3.9x |        3.2x |
-| 10 requests × 1s                  |           **6.0x** |              4.0x |        5.5x |
-| 20 requests × 1s (queued, conn=5) |          **14.5x** |             14.7x |        3.5x |
+| 4 requests × 1s                   |           **3.5x** |              3.5x |        3.0x |
+| 10 requests × 1s                  |          **10.3x** |             10.4x |        9.2x |
+| 20 requests × 1s (queued, conn=5) |           **6.1x** |              5.2x |        6.0x |
 
 ### Memory & allocations per request batch (lower is better)
 
 | Scenario                        | fast_curl (thread) allocated | fast_curl (fiber) allocated | Async::HTTP allocated |
 |---------------------------------|-----------------------------:|----------------------------:|----------------------:|
-| 4 requests × 1s                 |                 **278 obj**  |                     350 obj |             2,433 obj |
-| 10 requests × 1s                |                 **490 obj**  |                     756 obj |             4,763 obj |
-| 20 requests × 1s, conn=5        |                 **621 obj**  |                     750 obj |             8,536 obj |
-| 200 requests × 1s, conn=20      |               **5,188 obj**  |                   5,642 obj |            78,203 obj |
+| 4 requests × 1s                 |                 **125 obj**  |                     179 obj |             1,921 obj |
+| 10 requests × 1s                |                 **281 obj**  |                     335 obj |             3,344 obj |
+| 20 requests × 1s, conn=5        |                 **541 obj**  |                     595 obj |             5,898 obj |
+| 200 requests × 1s, conn=20      |               **5,222 obj**  |                   5,275 obj |            50,534 obj |
 
 Ruby heap delta stays near zero across all scenarios for fast_curl — most allocation
 happens in C, not on the Ruby heap.
@@ -263,7 +307,7 @@ happens in C, not on the Ruby heap.
 
 | Scenario                                                     |  Time |
 |--------------------------------------------------------------|------:|
-| 4 mixed requests (404, 500, DNS fail, 30s delay), timeout=2s | 4.02s |
+| 4 mixed requests (404, 500, DNS fail, 30s delay), timeout=2s | 0.68s |
 
 Bounded by `timeout=2s` rather than by the slow request.