fast_curl 0.2.0 → 0.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +49 -12
- data/ext/fast_curl/extconf.rb +4 -0
- data/ext/fast_curl/fast_curl.c +859 -475
- data/lib/fast_curl/version.rb +1 -1
- metadata +36 -26
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: ebf05021f9b72fb51019dd2a89a0ca91ef6e0f7c060f1447f137f245a5aac52e
|
|
4
|
+
data.tar.gz: e0f3d17b12a4ce18ce28cb832d594d7716719edc9513213348228b06ce12104d
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 2629435cd5203198a39697a831f1ff2a3fe71c41798b98fbe40654ad53954b5fbcf010e2e2a9fdb209932f045c50684868ed5c5d66e9c30a5226dfe6ca71910e
|
|
7
|
+
data.tar.gz: 576179e43d4203cf2bda1e3cb0cd19d58f4a98ad724a7c5ab8e1eaf8554f9f1294eba99cbe0f399bc64ba41e4ce60cc981be6a2b1409dcc600b0699c38562355
|
data/README.md
CHANGED
|
@@ -12,7 +12,9 @@ Ultra-fast parallel HTTP client for Ruby. C extension built on libcurl `curl_mul
|
|
|
12
12
|
|
|
13
13
|
## Installation
|
|
14
14
|
|
|
15
|
-
**Requirements**: Ruby >= 3.
|
|
15
|
+
**Requirements**: Ruby >= 3.1, libcurl
|
|
16
|
+
|
|
17
|
+
> **Why Ruby 3.1?** The C extension uses `rb_fiber_scheduler_current`, `rb_fiber_scheduler_block` and `rb_fiber_scheduler_unblock` to properly yield control to the Fiber Scheduler during I/O. These APIs are stable starting from Ruby 3.1. Without them, there is no correct way for a C extension to cooperate with the scheduler — earlier approaches (`rb_thread_schedule`) hold the GVL and block other fibers.
|
|
16
18
|
|
|
17
19
|
```ruby
|
|
18
20
|
gem 'fast_curl'
|
|
@@ -142,19 +144,54 @@ end
|
|
|
142
144
|
|
|
143
145
|
## Performance
|
|
144
146
|
|
|
145
|
-
|
|
147
|
+
Benchmarks against `httpbin.org`, 5 iterations with 1 warmup, median times.
|
|
148
|
+
Run yourself: `bundle exec ruby benchmark/local_bench.rb`.
|
|
149
|
+
|
|
150
|
+
Each request hits `/delay/1` (server-side 1-second delay), so sequential baseline
|
|
151
|
+
grows linearly while parallel clients stay near ~1s plus network overhead.
|
|
152
|
+
|
|
153
|
+
### Time to completion (lower is better)
|
|
154
|
+
|
|
155
|
+
| Scenario | Net::HTTP sequential | fast_curl (thread) | fast_curl (fiber/Async) | Async::HTTP::Internet |
|
|
156
|
+
|---------------------------------|---------------------:|-------------------:|------------------------:|----------------------:|
|
|
157
|
+
| 4 requests × 1s, conn=4 | 8.27s | 2.36s | 2.13s | 2.56s |
|
|
158
|
+
| 10 requests × 1s, conn=10 | 20.92s | 3.49s | 5.23s | 3.83s |
|
|
159
|
+
| 20 requests × 1s, conn=5 | 42.56s | 2.94s | 2.90s | 12.14s |
|
|
160
|
+
| 200 requests × 1s, conn=20 | — | 22.19s | 21.77s | 23.59s |
|
|
161
|
+
|
|
162
|
+
### Speedup vs Net::HTTP (median)
|
|
163
|
+
|
|
164
|
+
| Scenario | fast_curl (thread) | fast_curl (fiber) | Async::HTTP |
|
|
165
|
+
|-----------------------------------|-------------------:|------------------:|------------:|
|
|
166
|
+
| 4 requests × 1s | **3.5x** | 3.9x | 3.2x |
|
|
167
|
+
| 10 requests × 1s | **6.0x** | 4.0x | 5.5x |
|
|
168
|
+
| 20 requests × 1s (queued, conn=5) | **14.5x** | 14.7x | 3.5x |
|
|
169
|
+
|
|
170
|
+
### Memory & allocations per request batch (lower is better)
|
|
171
|
+
|
|
172
|
+
| Scenario | fast_curl (thread) allocated | fast_curl (fiber) allocated | Async::HTTP allocated |
|
|
173
|
+
|---------------------------------|-----------------------------:|----------------------------:|----------------------:|
|
|
174
|
+
| 4 requests × 1s | **278 obj** | 350 obj | 2,433 obj |
|
|
175
|
+
| 10 requests × 1s | **490 obj** | 756 obj | 4,763 obj |
|
|
176
|
+
| 20 requests × 1s, conn=5 | **621 obj** | 750 obj | 8,536 obj |
|
|
177
|
+
| 200 requests × 1s, conn=20 | **5,188 obj** | 5,642 obj | 78,203 obj |
|
|
178
|
+
|
|
179
|
+
Ruby heap delta stays near zero across all scenarios for fast_curl — most allocation
|
|
180
|
+
happens in C, not on the Ruby heap.
|
|
181
|
+
|
|
182
|
+
### Error handling
|
|
183
|
+
|
|
184
|
+
| Scenario | Time |
|
|
185
|
+
|--------------------------------------------------------------|------:|
|
|
186
|
+
| 4 mixed requests (404, 500, DNS fail, 30s delay), timeout=2s | 4.02s |
|
|
187
|
+
|
|
188
|
+
Bounded by `timeout=2s` rather than by the slow request.
|
|
146
189
|
|
|
147
|
-
|
|
148
|
-
|--------|------------|-------------|--------------|---------------|
|
|
149
|
-
| Net::HTTP sequential | 7.93s (+2.1 MB) | 24.20s (+0.3 MB) | 48.58s (+1.2 MB) | - |
|
|
150
|
-
| fast_curl (thread) | 2.09s (+0.7 MB) | 3.73s (+0.9 MB) | 3.76s (+0.0 MB) | 5.88s (+2.3 MB) |
|
|
151
|
-
| fast_curl (fiber) | 1.96s (+0.4 MB) | 4.86s (+0.0 MB) | 3.71s (+0.2 MB) | 9.60s (+1.6 MB) |
|
|
152
|
-
| Async::HTTP | 2.54s (+0.3 MB) | 4.27s (+0.4 MB) | 9.16s (+0.5 MB) | 22.44s (+10.7 MB) |
|
|
190
|
+
### Notes on the numbers
|
|
153
191
|
|
|
154
|
-
|
|
155
|
-
-
|
|
156
|
-
-
|
|
157
|
-
- Error handling (timeout=2s): 2.01s (+0.0 MB)
|
|
192
|
+
- **Net::HTTP sequential** is the proof-of-parallelism baseline — it confirms fast_curl and Async are actually running concurrently, not that they "beat" a different library. 4×1s sequentially = 4s, parallel = ~1s + overhead.
|
|
193
|
+
- **Variance is high against remote endpoints** (httpbin.org). For stable numbers, use `--local` which spawns a WEBrick server on 127.0.0.1.
|
|
194
|
+
- **fast_curl (thread) vs (fiber)**: same underlying C code, different scheduling. "thread" is the default; "fiber" kicks in automatically when called inside `Async do ... end`.
|
|
158
195
|
|
|
159
196
|
## License
|
|
160
197
|
|
data/ext/fast_curl/extconf.rb
CHANGED
|
@@ -6,7 +6,11 @@ abort "curl/curl.h is required" unless have_header("curl/curl.h")
|
|
|
6
6
|
have_header("ruby/thread.h")
|
|
7
7
|
have_header("ruby/fiber/scheduler.h")
|
|
8
8
|
|
|
9
|
+
have_func("curl_multi_wakeup", "curl/curl.h")
|
|
9
10
|
have_func("rb_fiber_scheduler_current", "ruby.h")
|
|
11
|
+
have_func("rb_fiber_scheduler_block", "ruby.h")
|
|
12
|
+
have_func("rb_fiber_scheduler_unblock", "ruby.h")
|
|
13
|
+
have_func("rb_fiber_current", "ruby.h")
|
|
10
14
|
have_func("rb_io_wait", "ruby.h")
|
|
11
15
|
|
|
12
16
|
$CFLAGS << " -std=c99 -O2 -Wall -Wextra -Wno-unused-parameter"
|