raptor 0.22.2 → 0.23.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +35 -0
- data/Dockerfile +3 -1
- data/README.md +38 -31
- data/docs/raptor-vs-puma.md +44 -38
- data/docs/talk.md +1295 -0
- data/ext/raptor_bpf/reuseport_select.bpf.c +18 -9
- data/ext/raptor_http/extconf.rb +2 -0
- data/ext/raptor_http/raptor_http.c +91 -119
- data/ext/raptor_http2/raptor_http2.c +272 -223
- data/ext/raptor_native/raptor_native.c +38 -20
- data/lib/rackup/handler/raptor.rb +12 -11
- data/lib/raptor/binder.rb +12 -12
- data/lib/raptor/cli.rb +18 -18
- data/lib/raptor/cluster.rb +44 -44
- data/lib/raptor/control_server.rb +3 -2
- data/lib/raptor/detached_body.rb +6 -2
- data/lib/raptor/http.rb +7 -5
- data/lib/raptor/http1.rb +233 -208
- data/lib/raptor/http2.rb +314 -267
- data/lib/raptor/reactor.rb +139 -149
- data/lib/raptor/reuseport_bpf.rb +62 -38
- data/lib/raptor/server.rb +53 -38
- data/lib/raptor/stats.rb +15 -3
- data/lib/raptor/thread_locals.rb +4 -3
- data/lib/raptor/version.rb +1 -1
- data/sig/generated/rackup/handler/raptor.rbs +2 -1
- data/sig/generated/raptor/binder.rbs +9 -9
- data/sig/generated/raptor/cli.rbs +6 -6
- data/sig/generated/raptor/cluster.rbs +9 -9
- data/sig/generated/raptor/control_server.rbs +2 -1
- data/sig/generated/raptor/detached_body.rbs +5 -1
- data/sig/generated/raptor/http.rbs +3 -3
- data/sig/generated/raptor/http1.rbs +100 -91
- data/sig/generated/raptor/http2.rbs +200 -189
- data/sig/generated/raptor/reactor.rbs +107 -118
- data/sig/generated/raptor/reuseport_bpf.rbs +54 -17
- data/sig/generated/raptor/server.rbs +26 -26
- data/sig/generated/raptor/stats.rbs +3 -2
- data/sig/generated/raptor/thread_locals.rbs +4 -3
- metadata +2 -2
- data/docs/brisrails-talk.md +0 -2398
data/docs/brisrails-talk.md
DELETED
|
@@ -1,2398 +0,0 @@
|
|
|
1
|
-
# Raptor
|
|
2
|
-
|
|
3
|
-
## A Ruby 4 Web Server
|
|
4
|
-
|
|
5
|
-
<img src="https://github.com/joshuay03.png" width="140" alt="Joshua Young" />
|
|
6
|
-
|
|
7
|
-
_Joshua Young ([`@joshuay03`](https://github.com/joshuay03)) · Senior Software Engineer at [Buildkite](https://buildkite.com/)_
|
|
8
|
-
|
|
9
|
-
_On the [Rails](https://github.com/rails/rails), [Puma](https://github.com/puma/puma), and [Concurrent Ruby](https://github.com/ruby-concurrency/concurrent-ruby) maintainer teams_
|
|
10
|
-
|
|
11
|
-
_BrisRails · August 2026_
|
|
12
|
-
|
|
13
|
-
_[luma.com/t3l24s8h](https://luma.com/t3l24s8h)_
|
|
14
|
-
|
|
15
|
-
<br>
|
|
16
|
-
<br>
|
|
17
|
-
<br>
|
|
18
|
-
<br>
|
|
19
|
-
<br>
|
|
20
|
-
<br>
|
|
21
|
-
<br>
|
|
22
|
-
<br>
|
|
23
|
-
|
|
24
|
-
---
|
|
25
|
-
|
|
26
|
-
<br>
|
|
27
|
-
<br>
|
|
28
|
-
<br>
|
|
29
|
-
<br>
|
|
30
|
-
<br>
|
|
31
|
-
<br>
|
|
32
|
-
<br>
|
|
33
|
-
<br>
|
|
34
|
-
|
|
35
|
-
## Hi
|
|
36
|
-
|
|
37
|
-
- <big>I built a Ruby web server over the last few months</big>
|
|
38
|
-
- <big>It's called Raptor</big>
|
|
39
|
-
- <big>It runs Rack apps, like Puma and Falcon do</big>
|
|
40
|
-
- <big>It's built around a Ractor-parallel HTTP parser</big>
|
|
41
|
-
- <big>It's fast enough that the numbers are interesting</big>
|
|
42
|
-
- <big>It's not production-ready. Nobody's Rails app should be behind it yet.</big>
|
|
43
|
-
|
|
44
|
-
<br>
|
|
45
|
-
<br>
|
|
46
|
-
<br>
|
|
47
|
-
<br>
|
|
48
|
-
<br>
|
|
49
|
-
<br>
|
|
50
|
-
<br>
|
|
51
|
-
<br>
|
|
52
|
-
|
|
53
|
-
---
|
|
54
|
-
|
|
55
|
-
<br>
|
|
56
|
-
<br>
|
|
57
|
-
<br>
|
|
58
|
-
<br>
|
|
59
|
-
<br>
|
|
60
|
-
<br>
|
|
61
|
-
<br>
|
|
62
|
-
<br>
|
|
63
|
-
|
|
64
|
-
## What this talk is
|
|
65
|
-
|
|
66
|
-
- <big>A tour of what's inside Raptor</big>
|
|
67
|
-
- <big>Why I built it</big>
|
|
68
|
-
- <big>What I got to learn along the way</big>
|
|
69
|
-
- <big>A lot of technology Ruby people don't usually get to touch:</big>
|
|
70
|
-
- Ractors, and what changed in Ruby 4
|
|
71
|
-
- Compare-and-swap and lock-free data structures
|
|
72
|
-
- Red-black trees
|
|
73
|
-
- Anonymous shared memory via mmap
|
|
74
|
-
- eBPF programs running inside the Linux kernel
|
|
75
|
-
- <big>Ruby APIs, native extensions, and one tiny BPF program, all working together in one server.</big>
|
|
76
|
-
|
|
77
|
-
<br>
|
|
78
|
-
<br>
|
|
79
|
-
<br>
|
|
80
|
-
<br>
|
|
81
|
-
<br>
|
|
82
|
-
<br>
|
|
83
|
-
<br>
|
|
84
|
-
<br>
|
|
85
|
-
|
|
86
|
-
---
|
|
87
|
-
|
|
88
|
-
<br>
|
|
89
|
-
<br>
|
|
90
|
-
<br>
|
|
91
|
-
<br>
|
|
92
|
-
<br>
|
|
93
|
-
<br>
|
|
94
|
-
<br>
|
|
95
|
-
<br>
|
|
96
|
-
|
|
97
|
-
## What's a web server, really
|
|
98
|
-
|
|
99
|
-
A web server is one process doing all of this:
|
|
100
|
-
|
|
101
|
-
- <big>Listens on a TCP port</big>
|
|
102
|
-
- <big>Accepts connections</big>
|
|
103
|
-
- <big>Reads bytes</big>
|
|
104
|
-
- <big>Parses those bytes into an HTTP request</big>
|
|
105
|
-
- <big>Hands the request to your Rack app</big>
|
|
106
|
-
- <big>Waits for a response</big>
|
|
107
|
-
- <big>Writes the response bytes back to the socket</big>
|
|
108
|
-
- <big>Closes the connection, or holds it open for another request</big>
|
|
109
|
-
|
|
110
|
-
Rails is not a web server. [Rack](https://github.com/rack/rack) is not a web server. Rack is a contract between web servers and Ruby apps.
|
|
111
|
-
|
|
112
|
-
<br>
|
|
113
|
-
<br>
|
|
114
|
-
<br>
|
|
115
|
-
<br>
|
|
116
|
-
<br>
|
|
117
|
-
<br>
|
|
118
|
-
<br>
|
|
119
|
-
<br>
|
|
120
|
-
|
|
121
|
-
---
|
|
122
|
-
|
|
123
|
-
<br>
|
|
124
|
-
<br>
|
|
125
|
-
<br>
|
|
126
|
-
<br>
|
|
127
|
-
<br>
|
|
128
|
-
<br>
|
|
129
|
-
<br>
|
|
130
|
-
<br>
|
|
131
|
-
|
|
132
|
-
## A Rack app
|
|
133
|
-
|
|
134
|
-
```ruby
|
|
135
|
-
# This is a complete Rack app. Nothing more is needed.
|
|
136
|
-
run ->(env) { [200, { "content-type" => "text/plain" }, ["hello\n"]] }
|
|
137
|
-
```
|
|
138
|
-
|
|
139
|
-
- <big>A callable object (something that responds to `#call`)</big>
|
|
140
|
-
- <big>Takes a `Hash` (the "Rack env")</big>
|
|
141
|
-
- <big>Returns `[status, headers, body]`</big>
|
|
142
|
-
|
|
143
|
-
That's the entire contract. Everything between the socket and that `#call` is the web server's problem.
|
|
144
|
-
|
|
145
|
-
<br>
|
|
146
|
-
<br>
|
|
147
|
-
<br>
|
|
148
|
-
<br>
|
|
149
|
-
<br>
|
|
150
|
-
<br>
|
|
151
|
-
<br>
|
|
152
|
-
<br>
|
|
153
|
-
|
|
154
|
-
---
|
|
155
|
-
|
|
156
|
-
<br>
|
|
157
|
-
<br>
|
|
158
|
-
<br>
|
|
159
|
-
<br>
|
|
160
|
-
<br>
|
|
161
|
-
<br>
|
|
162
|
-
<br>
|
|
163
|
-
<br>
|
|
164
|
-
|
|
165
|
-
## HTTP/1.1 in 60 seconds
|
|
166
|
-
|
|
167
|
-
For the folks who mostly know the Rails abstraction.
|
|
168
|
-
|
|
169
|
-
- <big>HTTP/1.1 is a text-based protocol</big>
|
|
170
|
-
- <big>One TCP connection carries one request at a time</big>
|
|
171
|
-
- <big>Keep-alive lets you send the next request on the same connection after the response comes back</big>
|
|
172
|
-
- <big>The request looks like this on the wire:</big>
|
|
173
|
-
|
|
174
|
-
```
|
|
175
|
-
GET /users/42 HTTP/1.1
|
|
176
|
-
Host: example.com
|
|
177
|
-
User-Agent: curl/8.0
|
|
178
|
-
Accept: application/json
|
|
179
|
-
|
|
180
|
-
```
|
|
181
|
-
|
|
182
|
-
- <big>The server parses those bytes into a `Hash` of headers and metadata</big>
|
|
183
|
-
- <big>Then writes a response in the same shape:</big>
|
|
184
|
-
|
|
185
|
-
```
|
|
186
|
-
HTTP/1.1 200 OK
|
|
187
|
-
Content-Type: application/json
|
|
188
|
-
Content-Length: 27
|
|
189
|
-
|
|
190
|
-
{"id":42,"name":"joshua"}
|
|
191
|
-
```
|
|
192
|
-
|
|
193
|
-
That's the entire protocol.
|
|
194
|
-
|
|
195
|
-
<br>
|
|
196
|
-
<br>
|
|
197
|
-
<br>
|
|
198
|
-
<br>
|
|
199
|
-
<br>
|
|
200
|
-
<br>
|
|
201
|
-
<br>
|
|
202
|
-
<br>
|
|
203
|
-
|
|
204
|
-
---
|
|
205
|
-
|
|
206
|
-
<br>
|
|
207
|
-
<br>
|
|
208
|
-
<br>
|
|
209
|
-
<br>
|
|
210
|
-
<br>
|
|
211
|
-
<br>
|
|
212
|
-
<br>
|
|
213
|
-
<br>
|
|
214
|
-
|
|
215
|
-
## Client and server, high-level
|
|
216
|
-
|
|
217
|
-
```mermaid
|
|
218
|
-
sequenceDiagram
|
|
219
|
-
participant C as Client
|
|
220
|
-
participant S as Server
|
|
221
|
-
|
|
222
|
-
C->>S: TCP SYN
|
|
223
|
-
S->>C: SYN-ACK
|
|
224
|
-
C->>S: ACK<br/>(connection open)
|
|
225
|
-
C->>S: GET /users/42 HTTP/1.1<br/>Host: ...
|
|
226
|
-
Note over S: Parse, call app, write response
|
|
227
|
-
S->>C: HTTP/1.1 200 OK<br/>Content-Type: ...<br/>body
|
|
228
|
-
C->>S: FIN (or next request on keep-alive)
|
|
229
|
-
```
|
|
230
|
-
|
|
231
|
-
- <big>Set up a TCP connection</big>
|
|
232
|
-
- <big>Send the request</big>
|
|
233
|
-
- <big>Get the response</big>
|
|
234
|
-
- <big>Close, or send another</big>
|
|
235
|
-
|
|
236
|
-
That is all a web server has to do, per request.
|
|
237
|
-
|
|
238
|
-
One thing to note: the whole `SYN` → `SYN-ACK` → `ACK` handshake at the top is handled entirely by the kernel. The web server doesn't do any of that. It calls `accept()`, and the kernel hands it a socket that's already connected.
|
|
239
|
-
|
|
240
|
-
- <big>Kernel does the handshake and puts the completed connection on the listener's accept queue</big>
|
|
241
|
-
- <big>Server calls `accept_nonblock` to pull the next connection off the queue</big>
|
|
242
|
-
- <big>For Raptor's plain-TCP listener, the BPF program we'll see later can choose which worker socket receives each new connection</big>
|
|
243
|
-
|
|
244
|
-
<br>
|
|
245
|
-
<br>
|
|
246
|
-
<br>
|
|
247
|
-
<br>
|
|
248
|
-
<br>
|
|
249
|
-
<br>
|
|
250
|
-
<br>
|
|
251
|
-
<br>
|
|
252
|
-
|
|
253
|
-
---
|
|
254
|
-
|
|
255
|
-
<br>
|
|
256
|
-
<br>
|
|
257
|
-
<br>
|
|
258
|
-
<br>
|
|
259
|
-
<br>
|
|
260
|
-
<br>
|
|
261
|
-
<br>
|
|
262
|
-
<br>
|
|
263
|
-
|
|
264
|
-
## The two servers you already know
|
|
265
|
-
|
|
266
|
-
**[Puma](https://github.com/puma/puma)** (since 2011)
|
|
267
|
-
|
|
268
|
-
- <big>Multi-threaded, one process per worker</big>
|
|
269
|
-
- <big>One thread pool per worker</big>
|
|
270
|
-
- <big>Uses `Mutex` + `ConditionVariable` for the pool</big>
|
|
271
|
-
- <big>HTTP/1.1 only</big>
|
|
272
|
-
- <big>Battle-tested. It's the safe pick.</big>
|
|
273
|
-
|
|
274
|
-
**[Falcon](https://github.com/socketry/falcon)**
|
|
275
|
-
|
|
276
|
-
- <big>Lightweight async tasks backed by fibers instead of a fixed app thread pool</big>
|
|
277
|
-
- Fibers are cooperatively scheduled coroutines and much cheaper to create than OS threads
|
|
278
|
-
- <big>Built on the [`async`](https://github.com/socketry/async) gem, mostly written by Samuel Williams ([`@ioquatix`](https://github.com/ioquatix))</big>
|
|
279
|
-
- <big>Speaks HTTP/2 natively</big>
|
|
280
|
-
- <big>Excellent at long-lived connections and streaming</big>
|
|
281
|
-
- <big>Works best when the whole app uses fiber-aware I/O (like `Async::HTTP::Client` for outgoing calls)</big>
|
|
282
|
-
|
|
283
|
-
Both are great servers. Both were designed and mostly written before Ractors existed.
|
|
284
|
-
|
|
285
|
-
<br>
|
|
286
|
-
<br>
|
|
287
|
-
<br>
|
|
288
|
-
<br>
|
|
289
|
-
<br>
|
|
290
|
-
<br>
|
|
291
|
-
<br>
|
|
292
|
-
<br>
|
|
293
|
-
|
|
294
|
-
---
|
|
295
|
-
|
|
296
|
-
<br>
|
|
297
|
-
<br>
|
|
298
|
-
<br>
|
|
299
|
-
<br>
|
|
300
|
-
<br>
|
|
301
|
-
<br>
|
|
302
|
-
<br>
|
|
303
|
-
<br>
|
|
304
|
-
|
|
305
|
-
## The Ruby 4 opening
|
|
306
|
-
|
|
307
|
-
- <big>Ruby 4 shipped last December</big>
|
|
308
|
-
- <big>Everyone talks about `it` as a block parameter</big>
|
|
309
|
-
- <big>The one that made me want to build a web server: **`Ractor::Port` stabilised**</big>
|
|
310
|
-
- `Ractor::Port` is a message channel that any number of Ractors can send to and one receiver drains from
|
|
311
|
-
- It's the primitive you need to build a pipeline where many workers produce results and one collector consumes them
|
|
312
|
-
- Before Ruby 4 it was experimental and the API kept changing
|
|
313
|
-
- <big>Ractors themselves were experimental for four years</big>
|
|
314
|
-
- API drifted between minor versions
|
|
315
|
-
- Ergonomics were rough
|
|
316
|
-
- Most Rubyists learned "Ractors exist" and then never touched them
|
|
317
|
-
- <big>In Ruby 4 the API has settled</big>
|
|
318
|
-
- <big>You can finally build against them without expecting them to break</big>
|
|
319
|
-
|
|
320
|
-
<br>
|
|
321
|
-
<br>
|
|
322
|
-
<br>
|
|
323
|
-
<br>
|
|
324
|
-
<br>
|
|
325
|
-
<br>
|
|
326
|
-
<br>
|
|
327
|
-
<br>
|
|
328
|
-
|
|
329
|
-
---
|
|
330
|
-
|
|
331
|
-
<br>
|
|
332
|
-
<br>
|
|
333
|
-
<br>
|
|
334
|
-
<br>
|
|
335
|
-
<br>
|
|
336
|
-
<br>
|
|
337
|
-
<br>
|
|
338
|
-
<br>
|
|
339
|
-
|
|
340
|
-
## The question I wanted to answer
|
|
341
|
-
|
|
342
|
-
What would a Ruby web server look like
|
|
343
|
-
|
|
344
|
-
- <big>if you took Ractors seriously</big>
|
|
345
|
-
- <big>instead of pretending the GVL was a fact of nature</big>
|
|
346
|
-
|
|
347
|
-
That's the pitch. The rest of the talk is how it went.
|
|
348
|
-
|
|
349
|
-
<br>
|
|
350
|
-
<br>
|
|
351
|
-
<br>
|
|
352
|
-
<br>
|
|
353
|
-
<br>
|
|
354
|
-
<br>
|
|
355
|
-
<br>
|
|
356
|
-
<br>
|
|
357
|
-
|
|
358
|
-
---
|
|
359
|
-
|
|
360
|
-
<br>
|
|
361
|
-
<br>
|
|
362
|
-
<br>
|
|
363
|
-
<br>
|
|
364
|
-
<br>
|
|
365
|
-
<br>
|
|
366
|
-
<br>
|
|
367
|
-
<br>
|
|
368
|
-
|
|
369
|
-
## Ractors 101
|
|
370
|
-
|
|
371
|
-
A Ractor is Ruby's answer to true parallelism.
|
|
372
|
-
|
|
373
|
-
- <big>Multiple Ractors run at the same time</big>
|
|
374
|
-
- <big>On different OS threads</big>
|
|
375
|
-
- <big>Each Ractor has its **own GVL**</big>
|
|
376
|
-
|
|
377
|
-
That last point is the whole thing.
|
|
378
|
-
|
|
379
|
-
- <big>Ruby's GVL is not process-wide</big>
|
|
380
|
-
- <big>It's Ractor-scoped</big>
|
|
381
|
-
- <big>Two Ractors can run Ruby code on two CPU cores at the same instant</big>
|
|
382
|
-
- <big>Actual parallelism, not just concurrency</big>
|
|
383
|
-
|
|
384
|
-
<br>
|
|
385
|
-
<br>
|
|
386
|
-
<br>
|
|
387
|
-
<br>
|
|
388
|
-
<br>
|
|
389
|
-
<br>
|
|
390
|
-
<br>
|
|
391
|
-
<br>
|
|
392
|
-
|
|
393
|
-
---
|
|
394
|
-
|
|
395
|
-
<br>
|
|
396
|
-
<br>
|
|
397
|
-
<br>
|
|
398
|
-
<br>
|
|
399
|
-
<br>
|
|
400
|
-
<br>
|
|
401
|
-
<br>
|
|
402
|
-
<br>
|
|
403
|
-
|
|
404
|
-
## The catch
|
|
405
|
-
|
|
406
|
-
Ractors are strict about what they can share.
|
|
407
|
-
|
|
408
|
-
```ruby
|
|
409
|
-
data = { name: "hello" }
|
|
410
|
-
|
|
411
|
-
Ractor.new do
|
|
412
|
-
puts data
|
|
413
|
-
end
|
|
414
|
-
# ArgumentError: can not isolate a Proc because it accesses outer variables (data)
|
|
415
|
-
```
|
|
416
|
-
|
|
417
|
-
The Ractor's block is isolated. It cannot reach back into the outer scope. If you want the Ractor to see `data`, you have to pass it explicitly, and even then only certain values can cross:
|
|
418
|
-
|
|
419
|
-
- <big>Deeply frozen objects (`Ractor.make_shareable(hash)`)</big>
|
|
420
|
-
- <big>Immutable value types (`Integer`, `Symbol`, `true`, `false`, `nil`)</big>
|
|
421
|
-
- <big>Values sent as messages between Ractors (deep-copied by default)</big>
|
|
422
|
-
- <big>Values sent with `move: true` (ownership transferred, sender loses access)</big>
|
|
423
|
-
- The runtime walks the object graph (the value and everything it references)
|
|
424
|
-
- Transfers ownership of every reachable object to the receiving Ractor
|
|
425
|
-
- The sender's references are replaced with a "moved" placeholder
|
|
426
|
-
- Any subsequent access from the sender raises `Ractor::MovedError`
|
|
427
|
-
- Cheaper than deep-copying for big object graphs, since the objects themselves aren't reallocated
|
|
428
|
-
|
|
429
|
-
Why so strict? Because shared mutable state plus true parallelism is where every data race bug lives. Ruby sidesteps the problem by refusing to let mutable state cross the boundary at all.
|
|
430
|
-
|
|
431
|
-
<br>
|
|
432
|
-
<br>
|
|
433
|
-
<br>
|
|
434
|
-
<br>
|
|
435
|
-
<br>
|
|
436
|
-
<br>
|
|
437
|
-
<br>
|
|
438
|
-
<br>
|
|
439
|
-
|
|
440
|
-
---
|
|
441
|
-
|
|
442
|
-
<br>
|
|
443
|
-
<br>
|
|
444
|
-
<br>
|
|
445
|
-
<br>
|
|
446
|
-
<br>
|
|
447
|
-
<br>
|
|
448
|
-
<br>
|
|
449
|
-
<br>
|
|
450
|
-
|
|
451
|
-
## Sending messages
|
|
452
|
-
|
|
453
|
-
```ruby
|
|
454
|
-
port = Ractor::Port.new
|
|
455
|
-
|
|
456
|
-
worker = Ractor.new(port) do |port|
|
|
457
|
-
loop do
|
|
458
|
-
data = Ractor.receive
|
|
459
|
-
port.send("processed: #{data}")
|
|
460
|
-
end
|
|
461
|
-
end
|
|
462
|
-
|
|
463
|
-
worker.send("hello")
|
|
464
|
-
puts port.receive # "processed: hello"
|
|
465
|
-
worker.send("world")
|
|
466
|
-
puts port.receive # "processed: world"
|
|
467
|
-
```
|
|
468
|
-
|
|
469
|
-
- <big>One Ractor sends work in via `Ractor.send`</big>
|
|
470
|
-
- <big>The Ractor block receives it via `Ractor.receive`</big>
|
|
471
|
-
- <big>Results go out through a `Ractor::Port`, which any number of Ractors can send to</big>
|
|
472
|
-
- <big>Ruby 4 is the first release where `Ractor::Port` is stable</big>
|
|
473
|
-
- <big>Every message crosses cleanly. Shareable values pass by reference, mutable values are deep-copied, and `move: true` transfers ownership.</big>
|
|
474
|
-
|
|
475
|
-
<br>
|
|
476
|
-
<br>
|
|
477
|
-
<br>
|
|
478
|
-
<br>
|
|
479
|
-
<br>
|
|
480
|
-
<br>
|
|
481
|
-
<br>
|
|
482
|
-
<br>
|
|
483
|
-
|
|
484
|
-
---
|
|
485
|
-
|
|
486
|
-
<br>
|
|
487
|
-
<br>
|
|
488
|
-
<br>
|
|
489
|
-
<br>
|
|
490
|
-
<br>
|
|
491
|
-
<br>
|
|
492
|
-
<br>
|
|
493
|
-
<br>
|
|
494
|
-
|
|
495
|
-
## The lesson
|
|
496
|
-
|
|
497
|
-
- <big>Ractors are **not** a drop-in "make it parallel" for existing code</big>
|
|
498
|
-
- <big>Almost no gem you use is Ractor-safe</big>
|
|
499
|
-
- <big>What Ractors are actually good for:</big>
|
|
500
|
-
- Small, well-scoped chunks of CPU work
|
|
501
|
-
- No global state to reach into
|
|
502
|
-
- No gem coupling
|
|
503
|
-
- A single hand-off boundary
|
|
504
|
-
|
|
505
|
-
And that description happens to fit HTTP parsing exactly.
|
|
506
|
-
|
|
507
|
-
<br>
|
|
508
|
-
<br>
|
|
509
|
-
<br>
|
|
510
|
-
<br>
|
|
511
|
-
<br>
|
|
512
|
-
<br>
|
|
513
|
-
<br>
|
|
514
|
-
<br>
|
|
515
|
-
|
|
516
|
-
---
|
|
517
|
-
|
|
518
|
-
<br>
|
|
519
|
-
<br>
|
|
520
|
-
<br>
|
|
521
|
-
<br>
|
|
522
|
-
<br>
|
|
523
|
-
<br>
|
|
524
|
-
<br>
|
|
525
|
-
<br>
|
|
526
|
-
|
|
527
|
-
## The idea
|
|
528
|
-
|
|
529
|
-
- <big>Puma parses HTTP on the app thread</big>
|
|
530
|
-
- <big>Same thread that's about to run your Rack app</big>
|
|
531
|
-
- <big>Both fight for the same one GVL</big>
|
|
532
|
-
- <big>Parsing time is stolen from your app</big>
|
|
533
|
-
|
|
534
|
-
If parsing runs in a Ractor:
|
|
535
|
-
|
|
536
|
-
- <big>It uses a **different GVL** than the app</big>
|
|
537
|
-
- <big>On a two-core machine, the parser and the app can run at the exact same instant</big>
|
|
538
|
-
- <big>Actual protocol and app parallelism inside one worker process</big>
|
|
539
|
-
|
|
540
|
-
That's the whole idea. Raptor uses that pipeline when a connection needs the reactor; complete requests on its eager paths parse inline instead of paying for a Ractor handoff.
|
|
541
|
-
|
|
542
|
-
<br>
|
|
543
|
-
<br>
|
|
544
|
-
<br>
|
|
545
|
-
<br>
|
|
546
|
-
<br>
|
|
547
|
-
<br>
|
|
548
|
-
<br>
|
|
549
|
-
<br>
|
|
550
|
-
|
|
551
|
-
---
|
|
552
|
-
|
|
553
|
-
<br>
|
|
554
|
-
<br>
|
|
555
|
-
<br>
|
|
556
|
-
<br>
|
|
557
|
-
<br>
|
|
558
|
-
<br>
|
|
559
|
-
<br>
|
|
560
|
-
<br>
|
|
561
|
-
|
|
562
|
-
## Raptor at a glance
|
|
563
|
-
|
|
564
|
-
```mermaid
|
|
565
|
-
flowchart TB
|
|
566
|
-
subgraph Master["Master process"]
|
|
567
|
-
MM["Watches workers<br/>Handles signals<br/>Reads shared mmap"]
|
|
568
|
-
end
|
|
569
|
-
|
|
570
|
-
subgraph Worker["Worker process, one of N"]
|
|
571
|
-
SRV["Server thread<br/>IO.select + accept_nonblock"]
|
|
572
|
-
RCT["Reactor thread<br/>NIO::Selector<br/>red-black tree of timeouts"]
|
|
573
|
-
|
|
574
|
-
subgraph RP1["HTTP/1.1 Ractor pool"]
|
|
575
|
-
RW1["Pipeline Ractor 1"]
|
|
576
|
-
RWM["Pipeline Ractor M"]
|
|
577
|
-
end
|
|
578
|
-
|
|
579
|
-
subgraph RP2["HTTP/2 Ractor pool"]
|
|
580
|
-
RW2["Pipeline Ractor 1"]
|
|
581
|
-
RWN["Pipeline Ractor N"]
|
|
582
|
-
end
|
|
583
|
-
|
|
584
|
-
COL["Collector threads<br/>drain Ractor::Ports"]
|
|
585
|
-
|
|
586
|
-
subgraph ATP["Thread pool, lock-free CAS queue"]
|
|
587
|
-
T1["App thread 1"]
|
|
588
|
-
T2["App thread 2"]
|
|
589
|
-
T3["App thread T"]
|
|
590
|
-
end
|
|
591
|
-
|
|
592
|
-
STA["Stats thread<br/>writes 1 Hz"]
|
|
593
|
-
|
|
594
|
-
SRV -->|"complete first read"| ATP
|
|
595
|
-
SRV --> RCT
|
|
596
|
-
RCT --> RP1
|
|
597
|
-
RCT --> RP2
|
|
598
|
-
RP1 --> COL
|
|
599
|
-
RP2 --> COL
|
|
600
|
-
COL --> ATP
|
|
601
|
-
STA --> SHM[("mmap")]
|
|
602
|
-
end
|
|
603
|
-
|
|
604
|
-
MM -->|"fork"| Worker
|
|
605
|
-
MM -.->|"reads"| SHM
|
|
606
|
-
```
|
|
607
|
-
|
|
608
|
-
Every part earns its keep. Let's walk it.
|
|
609
|
-
|
|
610
|
-
<br>
|
|
611
|
-
<br>
|
|
612
|
-
<br>
|
|
613
|
-
<br>
|
|
614
|
-
<br>
|
|
615
|
-
<br>
|
|
616
|
-
<br>
|
|
617
|
-
<br>
|
|
618
|
-
|
|
619
|
-
---
|
|
620
|
-
|
|
621
|
-
<br>
|
|
622
|
-
<br>
|
|
623
|
-
<br>
|
|
624
|
-
<br>
|
|
625
|
-
<br>
|
|
626
|
-
<br>
|
|
627
|
-
<br>
|
|
628
|
-
<br>
|
|
629
|
-
|
|
630
|
-
## The Ractor pools
|
|
631
|
-
|
|
632
|
-
Parsing runs in dedicated Ractor pools, one per HTTP protocol. This is where the parallelism actually kicks in.
|
|
633
|
-
|
|
634
|
-
```mermaid
|
|
635
|
-
flowchart LR
|
|
636
|
-
R["Reactor thread"] -->|"pool << state"| C["Coordinator"]
|
|
637
|
-
|
|
638
|
-
subgraph Pool
|
|
639
|
-
C
|
|
640
|
-
W1["Pipeline Ractor 1<br/>own GVL"]
|
|
641
|
-
W2["Pipeline Ractor 2<br/>own GVL"]
|
|
642
|
-
WN["Pipeline Ractor M<br/>own GVL"]
|
|
643
|
-
end
|
|
644
|
-
|
|
645
|
-
C -->|"forward to idle"| W1
|
|
646
|
-
W1 -->|"send self back when idle"| C
|
|
647
|
-
W1 -->|"result via Ractor::Port"| CL["Collector thread"]
|
|
648
|
-
CL --> ATP["App thread pool"]
|
|
649
|
-
```
|
|
650
|
-
|
|
651
|
-
<br>
|
|
652
|
-
<br>
|
|
653
|
-
<br>
|
|
654
|
-
<br>
|
|
655
|
-
<br>
|
|
656
|
-
<br>
|
|
657
|
-
<br>
|
|
658
|
-
<br>
|
|
659
|
-
|
|
660
|
-
---
|
|
661
|
-
|
|
662
|
-
<br>
|
|
663
|
-
<br>
|
|
664
|
-
<br>
|
|
665
|
-
<br>
|
|
666
|
-
<br>
|
|
667
|
-
<br>
|
|
668
|
-
<br>
|
|
669
|
-
<br>
|
|
670
|
-
|
|
671
|
-
## Coordinator-dispatch
|
|
672
|
-
|
|
673
|
-
- <big>One coordinator Ractor</big>
|
|
674
|
-
- <big>M pipeline Ractors</big>
|
|
675
|
-
- <big>When a pipeline Ractor becomes idle, it sends **itself** back to the coordinator:</big>
|
|
676
|
-
|
|
677
|
-
```ruby
|
|
678
|
-
coordinator.send(Ractor.current, move: true)
|
|
679
|
-
```
|
|
680
|
-
|
|
681
|
-
- <big>The coordinator always knows exactly which Ractors are free</big>
|
|
682
|
-
- <big>When work arrives, the coordinator forwards it straight to a waiting Ractor</big>
|
|
683
|
-
- <big>No queueing, no polling, no idle spin</big>
|
|
684
|
-
- <big>Results flow back through a shared `Ractor::Port` (the many-to-one channel that stabilised in Ruby 4)</big>
|
|
685
|
-
- <big>A Ruby-side collector thread drains the port and dispatches to the app thread pool</big>
|
|
686
|
-
|
|
687
|
-
I pulled the pattern into its own gem: **`ractor-pool`**.
|
|
688
|
-
|
|
689
|
-
<br>
|
|
690
|
-
<br>
|
|
691
|
-
<br>
|
|
692
|
-
<br>
|
|
693
|
-
<br>
|
|
694
|
-
<br>
|
|
695
|
-
<br>
|
|
696
|
-
<br>
|
|
697
|
-
|
|
698
|
-
---
|
|
699
|
-
|
|
700
|
-
<br>
|
|
701
|
-
<br>
|
|
702
|
-
<br>
|
|
703
|
-
<br>
|
|
704
|
-
<br>
|
|
705
|
-
<br>
|
|
706
|
-
<br>
|
|
707
|
-
<br>
|
|
708
|
-
|
|
709
|
-
## The parser itself
|
|
710
|
-
|
|
711
|
-
- <big>Native C extension ([Ragel](https://www.colm.net/open-source/ragel/)-generated finite-state machine)</big>
|
|
712
|
-
- Ragel is a tool that takes a state-machine spec and generates fast C parsing code from it
|
|
713
|
-
- <big>Declared Ractor-safe with `rb_ext_ractor_safe(true)`, a C API for extensions to tell the VM they're safe to call from any Ractor</big>
|
|
714
|
-
- <big>Holds no per-parser Ruby state in the extension</big>
|
|
715
|
-
- <big>Everything goes into the caller-supplied env `Hash`</big>
|
|
716
|
-
- <big>Interns the ~40 most common HTTP header keys once at load time</big>
|
|
717
|
-
- "Interning" means creating one canonical `String` object for a value and reusing it forever
|
|
718
|
-
- `HTTP_HOST`, `HTTP_USER_AGENT`, `HTTP_ACCEPT`, `CONTENT_LENGTH`, and so on
|
|
719
|
-
- <big>Every request's env `Hash` reuses the same `String` objects for its header names</big>
|
|
720
|
-
- <big>Skips per-request key allocation</big>
|
|
721
|
-
- <big>Lets Ruby's `Hash` use the interned `String`'s cached hash code</big>
|
|
722
|
-
|
|
723
|
-
<br>
|
|
724
|
-
<br>
|
|
725
|
-
<br>
|
|
726
|
-
<br>
|
|
727
|
-
<br>
|
|
728
|
-
<br>
|
|
729
|
-
<br>
|
|
730
|
-
<br>
|
|
731
|
-
|
|
732
|
-
---
|
|
733
|
-
|
|
734
|
-
<br>
|
|
735
|
-
<br>
|
|
736
|
-
<br>
|
|
737
|
-
<br>
|
|
738
|
-
<br>
|
|
739
|
-
<br>
|
|
740
|
-
<br>
|
|
741
|
-
<br>
|
|
742
|
-
|
|
743
|
-
## Two GVLs, in one picture
|
|
744
|
-
|
|
745
|
-
```mermaid
|
|
746
|
-
flowchart LR
|
|
747
|
-
subgraph GVL1["Main-process GVL"]
|
|
748
|
-
App1["App thread 1"]
|
|
749
|
-
App2["App thread 2"]
|
|
750
|
-
App3["App thread 3"]
|
|
751
|
-
end
|
|
752
|
-
|
|
753
|
-
subgraph GVL2["Pipeline Ractor GVL"]
|
|
754
|
-
Prs["Parser"]
|
|
755
|
-
end
|
|
756
|
-
|
|
757
|
-
App1 -.->|"waiting on GVL"| App2
|
|
758
|
-
Prs -.->|"parsing on a different CPU<br/>at the same instant"| App1
|
|
759
|
-
```
|
|
760
|
-
|
|
761
|
-
- <big>App threads share one GVL among themselves. Only one runs Ruby at a time.</big>
|
|
762
|
-
- <big>The pipeline Ractor has its own GVL. It runs Ruby independently.</big>
|
|
763
|
-
- <big>A pipeline Ractor and an app thread can run Ruby code at the same time, on two CPU cores</big>
|
|
764
|
-
- <big>That is not something you get out of the box in Ruby</big>
|
|
765
|
-
|
|
766
|
-
<br>
|
|
767
|
-
<br>
|
|
768
|
-
<br>
|
|
769
|
-
<br>
|
|
770
|
-
<br>
|
|
771
|
-
<br>
|
|
772
|
-
<br>
|
|
773
|
-
<br>
|
|
774
|
-
|
|
775
|
-
---
|
|
776
|
-
|
|
777
|
-
<br>
|
|
778
|
-
<br>
|
|
779
|
-
<br>
|
|
780
|
-
<br>
|
|
781
|
-
<br>
|
|
782
|
-
<br>
|
|
783
|
-
<br>
|
|
784
|
-
<br>
|
|
785
|
-
|
|
786
|
-
## The app thread pool
|
|
787
|
-
|
|
788
|
-
Where your Rack app runs, and where the response gets written back to the socket.
|
|
789
|
-
|
|
790
|
-
- <big>Puma's pool coordinates a Ruby `Queue` with `Mutex + ConditionVariable`</big>
|
|
791
|
-
- <big>Every enqueue and every dequeue takes the mutex</big>
|
|
792
|
-
- <big>Under low concurrency, it's fine</big>
|
|
793
|
-
- <big>Under contention, the mutex becomes a serialisation point</big>
|
|
794
|
-
- <big>All threads have to line up to touch the queue</big>
|
|
795
|
-
|
|
796
|
-
Raptor's pool is **lock-free on the hot path**.
|
|
797
|
-
|
|
798
|
-
To explain that, one minute on one atomic primitive.
|
|
799
|
-
|
|
800
|
-
<br>
|
|
801
|
-
<br>
|
|
802
|
-
<br>
|
|
803
|
-
<br>
|
|
804
|
-
<br>
|
|
805
|
-
<br>
|
|
806
|
-
<br>
|
|
807
|
-
<br>
|
|
808
|
-
|
|
809
|
-
---
|
|
810
|
-
|
|
811
|
-
<br>
|
|
812
|
-
<br>
|
|
813
|
-
<br>
|
|
814
|
-
<br>
|
|
815
|
-
<br>
|
|
816
|
-
<br>
|
|
817
|
-
<br>
|
|
818
|
-
<br>
|
|
819
|
-
|
|
820
|
-
## Compare-and-swap (CAS)
|
|
821
|
-
|
|
822
|
-
```mermaid
|
|
823
|
-
flowchart LR
|
|
824
|
-
subgraph Memory["Memory word"]
|
|
825
|
-
V["current: 42"]
|
|
826
|
-
end
|
|
827
|
-
|
|
828
|
-
T1["Thread A<br/>expected = 42<br/>new_value = 99"]
|
|
829
|
-
T2["Thread B<br/>expected = 42<br/>new_value = 77"]
|
|
830
|
-
|
|
831
|
-
T1 -->|"CAS(42, 99)<br/>succeeds"| Memory
|
|
832
|
-
T2 -->|"CAS(42, 77)<br/>fails because current is now 99"| Memory
|
|
833
|
-
```
|
|
834
|
-
|
|
835
|
-
- <big>A hardware-supported atomic operation</big>
|
|
836
|
-
- <big>Reads a memory word</big>
|
|
837
|
-
- <big>Compares to what you expected</big>
|
|
838
|
-
- <big>If they match, replaces with a new value</big>
|
|
839
|
-
- <big>All atomically. No lock.</big>
|
|
840
|
-
|
|
841
|
-
On x86 it is commonly `LOCK CMPXCHG`. On ARM it is commonly an exclusive load/store pair such as `LDXR` / `STXR`.
|
|
842
|
-
|
|
843
|
-
<br>
|
|
844
|
-
<br>
|
|
845
|
-
<br>
|
|
846
|
-
<br>
|
|
847
|
-
<br>
|
|
848
|
-
<br>
|
|
849
|
-
<br>
|
|
850
|
-
<br>
|
|
851
|
-
|
|
852
|
-
---
|
|
853
|
-
|
|
854
|
-
<br>
|
|
855
|
-
<br>
|
|
856
|
-
<br>
|
|
857
|
-
<br>
|
|
858
|
-
<br>
|
|
859
|
-
<br>
|
|
860
|
-
<br>
|
|
861
|
-
<br>
|
|
862
|
-
|
|
863
|
-
## What CAS buys you
|
|
864
|
-
|
|
865
|
-
```ruby
|
|
866
|
-
# If @slot == expected, set @slot = new_value.
|
|
867
|
-
# Returns true if the swap happened, false if @slot changed.
|
|
868
|
-
cas(@slot, expected, new_value)
|
|
869
|
-
```
|
|
870
|
-
|
|
871
|
-
- <big>If a state transition can be published with CAS, threads do not need a lock for that transition</big>
|
|
872
|
-
- <big>If the CAS fails, you retry (someone else got there first)</big>
|
|
873
|
-
- <big>Many high-performance concurrent data structures are built on this</big>
|
|
874
|
-
- Java's `ConcurrentHashMap`
|
|
875
|
-
- Rust's `AtomicUsize`
|
|
876
|
-
- Go's `sync/atomic`
|
|
877
|
-
- <big>[`concurrent-ruby`](https://github.com/ruby-concurrency/concurrent-ruby) has had CAS primitives (`AtomicReference`, `AtomicBoolean`, `AtomicFixnum`) since 2013</big>
|
|
878
|
-
- I could have built around those primitives, but Raptor also needed a lock-free FIFO and lock-free waiter parking
|
|
879
|
-
- I built that smaller, native set of pieces as `atomic-ruby`, with its C extension explicitly marked Ractor-safe
|
|
880
|
-
|
|
881
|
-
<br>
|
|
882
|
-
<br>
|
|
883
|
-
<br>
|
|
884
|
-
<br>
|
|
885
|
-
<br>
|
|
886
|
-
<br>
|
|
887
|
-
<br>
|
|
888
|
-
<br>
|
|
889
|
-
|
|
890
|
-
---
|
|
891
|
-
|
|
892
|
-
<br>
|
|
893
|
-
<br>
|
|
894
|
-
<br>
|
|
895
|
-
<br>
|
|
896
|
-
<br>
|
|
897
|
-
<br>
|
|
898
|
-
<br>
|
|
899
|
-
<br>
|
|
900
|
-
|
|
901
|
-
## CAS vs Mutex
|
|
902
|
-
|
|
903
|
-
**Mutex is the default for a reason**
|
|
904
|
-
|
|
905
|
-
- <big>Simple mental model: lock, do work, unlock</big>
|
|
906
|
-
- <big>Works for any-size critical section, touching any number of fields</big>
|
|
907
|
-
- <big>Fine at low contention</big>
|
|
908
|
-
|
|
909
|
-
But it has real costs:
|
|
910
|
-
|
|
911
|
-
- <big>Under contention, threads sleep and wait; the kernel context-switches them in and out</big>
|
|
912
|
-
- <big>Only one thread inside the critical section at a time, even for pure reads</big>
|
|
913
|
-
- <big>Priority inversion, and the risk of deadlock once you have more than one lock</big>
|
|
914
|
-
|
|
915
|
-
<br>
|
|
916
|
-
<br>
|
|
917
|
-
<br>
|
|
918
|
-
<br>
|
|
919
|
-
<br>
|
|
920
|
-
<br>
|
|
921
|
-
<br>
|
|
922
|
-
<br>
|
|
923
|
-
|
|
924
|
-
---
|
|
925
|
-
|
|
926
|
-
<br>
|
|
927
|
-
<br>
|
|
928
|
-
<br>
|
|
929
|
-
<br>
|
|
930
|
-
<br>
|
|
931
|
-
<br>
|
|
932
|
-
<br>
|
|
933
|
-
<br>
|
|
934
|
-
|
|
935
|
-
## When CAS is the better fit
|
|
936
|
-
|
|
937
|
-
CAS has its own costs too:
|
|
938
|
-
|
|
939
|
-
- <big>It only operates on one machine word at a time</big>
|
|
940
|
-
- A machine word is one aligned chunk of memory the CPU can read or write in one go: 8 bytes on 64-bit CPUs, 4 on 32-bit
|
|
941
|
-
- So one CAS can update one pointer, one integer, or one small value that fits in a word
|
|
942
|
-
- You can't atomically update two independent fields of a bigger object with a single CAS
|
|
943
|
-
- <big>Retry loops can burn CPU when contention is extreme</big>
|
|
944
|
-
- <big>Harder to reason about than a mutex block</big>
|
|
945
|
-
|
|
946
|
-
Where it pays off:
|
|
947
|
-
|
|
948
|
-
- <big>Single-slot updates, like a queue head or an atomic counter</big>
|
|
949
|
-
- <big>State that's read a lot more often than it's written</big>
|
|
950
|
-
- <big>Hot paths where a mutex would become the scalability wall</big>
|
|
951
|
-
|
|
952
|
-
Why we picked CAS for Raptor's thread pool:
|
|
953
|
-
|
|
954
|
-
- <big>The queue can publish head, tail, and link changes through small atomic steps</big>
|
|
955
|
-
- <big>The server thread reads pool metrics on every accept iteration, thousands of times per second</big>
|
|
956
|
-
- <big>Under a queue-wide mutex, readers and writers serialise around the same lock</big>
|
|
957
|
-
- <big>Atomic counters avoid that lock. They still cost a synchronised memory access, but they do not park another thread.</big>
|
|
958
|
-
|
|
959
|
-
<br>
|
|
960
|
-
<br>
|
|
961
|
-
<br>
|
|
962
|
-
<br>
|
|
963
|
-
<br>
|
|
964
|
-
<br>
|
|
965
|
-
<br>
|
|
966
|
-
<br>
|
|
967
|
-
|
|
968
|
-
---
|
|
969
|
-
|
|
970
|
-
<br>
|
|
971
|
-
<br>
|
|
972
|
-
<br>
|
|
973
|
-
<br>
|
|
974
|
-
<br>
|
|
975
|
-
<br>
|
|
976
|
-
<br>
|
|
977
|
-
<br>
|
|
978
|
-
|
|
979
|
-
## atomic-ruby
|
|
980
|
-
|
|
981
|
-
Another gem I wrote. Native C extension. Exposes CAS-based primitives to Ruby:
|
|
982
|
-
|
|
983
|
-
- <big>`Atom` – a CAS-protected reference cell holding one Ruby value</big>
|
|
984
|
-
- <big>`AtomicBoolean` – a type-specialised boolean</big>
|
|
985
|
-
- <big>`AtomicQueue` – a multi-producer, multi-consumer FIFO</big>
|
|
986
|
-
- <big>`AtomicThreadPool` – the thread pool Raptor uses</big>
|
|
987
|
-
- <big>`AtomicConditionVariable` – lock-free waiter parking</big>
|
|
988
|
-
|
|
989
|
-
Ruby's `VALUE` type on 64-bit is a tagged pointer. It fits in one machine word, so the native extension can compare and replace the reference atomically.
|
|
990
|
-
|
|
991
|
-
```c
|
|
992
|
-
old = ATOMIC_VALUE_CAS(&atom->value, expected, new_value);
|
|
993
|
-
```
|
|
994
|
-
|
|
995
|
-
<br>
|
|
996
|
-
<br>
|
|
997
|
-
<br>
|
|
998
|
-
<br>
|
|
999
|
-
<br>
|
|
1000
|
-
<br>
|
|
1001
|
-
<br>
|
|
1002
|
-
<br>
|
|
1003
|
-
|
|
1004
|
-
---
|
|
1005
|
-
|
|
1006
|
-
<br>
|
|
1007
|
-
<br>
|
|
1008
|
-
<br>
|
|
1009
|
-
<br>
|
|
1010
|
-
<br>
|
|
1011
|
-
<br>
|
|
1012
|
-
<br>
|
|
1013
|
-
<br>
|
|
1014
|
-
|
|
1015
|
-
## The Michael-Scott queue
|
|
1016
|
-
|
|
1017
|
-
Raptor's thread pool uses a lock-free linked FIFO from `atomic-ruby`:
|
|
1018
|
-
|
|
1019
|
-
```mermaid
|
|
1020
|
-
flowchart LR
|
|
1021
|
-
H["atomic head"] --> S["dummy sentinel"]
|
|
1022
|
-
S --> A["work A"]
|
|
1023
|
-
A --> B["work B"]
|
|
1024
|
-
B --> N["nil"]
|
|
1025
|
-
T["atomic tail"] --> B
|
|
1026
|
-
P["producer: work C"] -->|"CAS B.next from nil to C<br/>then advance tail"| C["work C"]
|
|
1027
|
-
Q["consumer"] -->|"read A<br/>CAS head from S to A"| H
|
|
1028
|
-
```
|
|
1029
|
-
|
|
1030
|
-
<br>
|
|
1031
|
-
<br>
|
|
1032
|
-
<br>
|
|
1033
|
-
<br>
|
|
1034
|
-
<br>
|
|
1035
|
-
<br>
|
|
1036
|
-
<br>
|
|
1037
|
-
<br>
|
|
1038
|
-
|
|
1039
|
-
---
|
|
1040
|
-
|
|
1041
|
-
<br>
|
|
1042
|
-
<br>
|
|
1043
|
-
<br>
|
|
1044
|
-
<br>
|
|
1045
|
-
<br>
|
|
1046
|
-
<br>
|
|
1047
|
-
<br>
|
|
1048
|
-
<br>
|
|
1049
|
-
|
|
1050
|
-
## Why this pattern works
|
|
1051
|
-
|
|
1052
|
-
- <big>A dummy node separates the queue's head position from its first value</big>
|
|
1053
|
-
- <big>Producers append at the tail; consumers advance the head</big>
|
|
1054
|
-
- <big>Multiple producers and consumers can make progress without one queue-wide lock</big>
|
|
1055
|
-
- <big>Push and pop are O(1), with CAS retries when another thread wins a race</big>
|
|
1056
|
-
- <big>Lock-free</big>
|
|
1057
|
-
- <big>It's the Michael-Scott queue, a classic concurrent FIFO design</big>
|
|
1058
|
-
|
|
1059
|
-
The bit that matters most in practice:
|
|
1060
|
-
|
|
1061
|
-
- <big>Queue length and active count are tracked in separate atoms</big>
|
|
1062
|
-
- <big>The server thread reads them every iteration of the accept loop</big>
|
|
1063
|
-
- <big>Those reads do not acquire the queue's mutation lock, because there isn't one</big>
|
|
1064
|
-
- <big>The values are point-in-time snapshots; they can change immediately after being read</big>
|
|
1065
|
-
|
|
1066
|
-
<br>
|
|
1067
|
-
<br>
|
|
1068
|
-
<br>
|
|
1069
|
-
<br>
|
|
1070
|
-
<br>
|
|
1071
|
-
<br>
|
|
1072
|
-
<br>
|
|
1073
|
-
<br>
|
|
1074
|
-
|
|
1075
|
-
---
|
|
1076
|
-
|
|
1077
|
-
<br>
|
|
1078
|
-
<br>
|
|
1079
|
-
<br>
|
|
1080
|
-
<br>
|
|
1081
|
-
<br>
|
|
1082
|
-
<br>
|
|
1083
|
-
<br>
|
|
1084
|
-
<br>
|
|
1085
|
-
|
|
1086
|
-
## Scaling app threads without making CPU contention worse
|
|
1087
|
-
|
|
1088
|
-
The pool starts at `threads` and scales automatically when more threads would help.
|
|
1089
|
-
|
|
1090
|
-
- <big>A native CRuby thread hook measures time running, blocked outside the GVL, and waiting for the GVL</big>
|
|
1091
|
-
- <big>The queue has to stay non-empty, and every current worker has to be active</big>
|
|
1092
|
-
- <big>Blocked time has to exceed half of worker time</big>
|
|
1093
|
-
- <big>GVL wait has to stay below two percent</big>
|
|
1094
|
-
- <big>Only then does the pool add a temporary thread</big>
|
|
1095
|
-
- <big>When the queue drains, temporary threads leave and the pool returns to `threads`</big>
|
|
1096
|
-
|
|
1097
|
-
The distinction matters. More threads help when requests are asleep in database or network calls. They make CPU-bound Ruby slower when the GVL is already the bottleneck.
|
|
1098
|
-
|
|
1099
|
-
Growth has no fixed limit by default. Set `max_threads` to cap it, or set it to `threads` to keep the pool fixed. OS threads still are not as cheap as fibers.
|
|
1100
|
-
|
|
1101
|
-
Raptor clears application thread locals when each request finishes. Running every request in a fresh Fiber is also available when an application needs Fiber-local isolation. Parser and response buffers are kept separately so the server can still reuse them without carrying application state into the next request.
|
|
1102
|
-
|
|
1103
|
-
<br>
|
|
1104
|
-
<br>
|
|
1105
|
-
<br>
|
|
1106
|
-
<br>
|
|
1107
|
-
<br>
|
|
1108
|
-
<br>
|
|
1109
|
-
<br>
|
|
1110
|
-
<br>
|
|
1111
|
-
|
|
1112
|
-
---
|
|
1113
|
-
|
|
1114
|
-
<br>
|
|
1115
|
-
<br>
|
|
1116
|
-
<br>
|
|
1117
|
-
<br>
|
|
1118
|
-
<br>
|
|
1119
|
-
<br>
|
|
1120
|
-
<br>
|
|
1121
|
-
<br>
|
|
1122
|
-
|
|
1123
|
-
## The reactor
|
|
1124
|
-
|
|
1125
|
-
- <big>Every network server has to track connections that are open but not yet ready to be read</big>
|
|
1126
|
-
- <big>Slow clients, half-sent requests, idle keep-alive sockets</big>
|
|
1127
|
-
- <big>That's the reactor's job</big>
|
|
1128
|
-
|
|
1129
|
-
Raptor's reactor runs `NIO::Selector`, from the [`nio4r`](https://github.com/socketry/nio4r) library.
|
|
1130
|
-
|
|
1131
|
-
- <big>`nio4r` is a Ruby wrapper around the operating system's efficient "wait on many sockets at once" APIs</big>
|
|
1132
|
-
- <big>On Linux it uses `epoll`, on macOS/BSD it uses `kqueue`</big>
|
|
1133
|
-
- <big>Puma uses the same library for the same purpose</big>
|
|
1134
|
-
|
|
1135
|
-
Where Raptor diverges from Puma is the data structure it uses for timeouts.
|
|
1136
|
-
|
|
1137
|
-
<br>
|
|
1138
|
-
<br>
|
|
1139
|
-
<br>
|
|
1140
|
-
<br>
|
|
1141
|
-
<br>
|
|
1142
|
-
<br>
|
|
1143
|
-
<br>
|
|
1144
|
-
<br>
|
|
1145
|
-
|
|
1146
|
-
---
|
|
1147
|
-
|
|
1148
|
-
<br>
|
|
1149
|
-
<br>
|
|
1150
|
-
<br>
|
|
1151
|
-
<br>
|
|
1152
|
-
<br>
|
|
1153
|
-
<br>
|
|
1154
|
-
<br>
|
|
1155
|
-
<br>
|
|
1156
|
-
|
|
1157
|
-
## Tracking timeouts
|
|
1158
|
-
|
|
1159
|
-
**Puma**
|
|
1160
|
-
|
|
1161
|
-
- <big>Plain Ruby `Array` of clients</big>
|
|
1162
|
-
- <big>Sorted by deadline after each batch of inserts</big>
|
|
1163
|
-
- <big>Sort is O(n log n)</big>
|
|
1164
|
-
- <big>Removing a specific client is a linear scan, O(n)</big>
|
|
1165
|
-
- <big>Fine at tens of connections. Linear at thousands.</big>
|
|
1166
|
-
|
|
1167
|
-
**Raptor**
|
|
1168
|
-
|
|
1169
|
-
- <big>Red-black tree</big>
|
|
1170
|
-
- <big>Insert: O(log n)</big>
|
|
1171
|
-
- <big>Remove by key: O(log n)</big>
|
|
1172
|
-
- <big>Walk earliest-first, break on the first non-expired node</big>
|
|
1173
|
-
|
|
1174
|
-
At 100 connections it's a wash. At 1000+, the difference is real.
|
|
1175
|
-
|
|
1176
|
-
<br>
|
|
1177
|
-
<br>
|
|
1178
|
-
<br>
|
|
1179
|
-
<br>
|
|
1180
|
-
<br>
|
|
1181
|
-
<br>
|
|
1182
|
-
<br>
|
|
1183
|
-
<br>
|
|
1184
|
-
|
|
1185
|
-
---
|
|
1186
|
-
|
|
1187
|
-
<br>
|
|
1188
|
-
<br>
|
|
1189
|
-
<br>
|
|
1190
|
-
<br>
|
|
1191
|
-
<br>
|
|
1192
|
-
<br>
|
|
1193
|
-
<br>
|
|
1194
|
-
<br>
|
|
1195
|
-
|
|
1196
|
-
## Red-black tree
|
|
1197
|
-
|
|
1198
|
-
A self-balancing binary search tree.
|
|
1199
|
-
|
|
1200
|
-
- <big>Every operation (insert, delete, lookup, min) is O(log n)</big>
|
|
1201
|
-
- <big>Five invariants that keep it balanced:</big>
|
|
1202
|
-
1. Every node is either red or black
|
|
1203
|
-
2. The root is black
|
|
1204
|
-
3. Red nodes have black children (no two reds in a row)
|
|
1205
|
-
4. Every path from a node to a leaf has the same number of black nodes
|
|
1206
|
-
5. Leaves are black
|
|
1207
|
-
- <big>Insertion adds a red node, then rotates and recolors to restore the invariants</big>
|
|
1208
|
-
|
|
1209
|
-
It's the data structure behind `TreeMap` in Java and `std::map` in C++.
|
|
1210
|
-
|
|
1211
|
-
<br>
|
|
1212
|
-
<br>
|
|
1213
|
-
<br>
|
|
1214
|
-
<br>
|
|
1215
|
-
<br>
|
|
1216
|
-
<br>
|
|
1217
|
-
<br>
|
|
1218
|
-
<br>
|
|
1219
|
-
|
|
1220
|
-
---
|
|
1221
|
-
|
|
1222
|
-
<br>
|
|
1223
|
-
<br>
|
|
1224
|
-
<br>
|
|
1225
|
-
<br>
|
|
1226
|
-
<br>
|
|
1227
|
-
<br>
|
|
1228
|
-
<br>
|
|
1229
|
-
<br>
|
|
1230
|
-
|
|
1231
|
-
## The reactor's tree
|
|
1232
|
-
|
|
1233
|
-
```mermaid
|
|
1234
|
-
flowchart TB
|
|
1235
|
-
Root["timeout_at = 100<br/>black"]
|
|
1236
|
-
L["timeout_at = 50<br/>red"]
|
|
1237
|
-
R["timeout_at = 150<br/>red"]
|
|
1238
|
-
LL["timeout_at = 25<br/>black"]
|
|
1239
|
-
LR["timeout_at = 75<br/>black"]
|
|
1240
|
-
|
|
1241
|
-
Root --> L
|
|
1242
|
-
Root --> R
|
|
1243
|
-
L --> LL
|
|
1244
|
-
L --> LR
|
|
1245
|
-
```
|
|
1246
|
-
|
|
1247
|
-
- <big>Each node is a `TimeoutClient < RedBlackTree::Node`</big>
|
|
1248
|
-
- <big>Ordered by `timeout_at` (a monotonic float)</big>
|
|
1249
|
-
- <big>Every read on a socket resets its deadline: delete + reinsert</big>
|
|
1250
|
-
- <big>That's what makes the tree beat a min-heap</big>
|
|
1251
|
-
- Heap peek is O(1), but arbitrary delete is O(n) because you have to find the node
|
|
1252
|
-
- Tree delete-by-key is O(log n)
|
|
1253
|
-
|
|
1254
|
-
<br>
|
|
1255
|
-
<br>
|
|
1256
|
-
<br>
|
|
1257
|
-
<br>
|
|
1258
|
-
<br>
|
|
1259
|
-
<br>
|
|
1260
|
-
<br>
|
|
1261
|
-
<br>
|
|
1262
|
-
|
|
1263
|
-
---
|
|
1264
|
-
|
|
1265
|
-
<br>
|
|
1266
|
-
<br>
|
|
1267
|
-
<br>
|
|
1268
|
-
<br>
|
|
1269
|
-
<br>
|
|
1270
|
-
<br>
|
|
1271
|
-
<br>
|
|
1272
|
-
<br>
|
|
1273
|
-
|
|
1274
|
-
## The three timeouts
|
|
1275
|
-
|
|
1276
|
-
- <big>**`first_data_timeout` (30s)** – applied to a fresh connection with no bytes read yet</big>
|
|
1277
|
-
- <big>**`chunk_data_timeout` (10s)** – applied once data has started arriving but the request is incomplete</big>
|
|
1278
|
-
- <big>**`persistent_data_timeout` (65s)** – applied to a keep-alive socket sitting idle between requests</big>
|
|
1279
|
-
|
|
1280
|
-
On timeout: write `HTTP/1.1 408 Request Timeout` and close.
|
|
1281
|
-
|
|
1282
|
-
I pulled the tree out into its own gem: **`red-black-tree`**.
|
|
1283
|
-
|
|
1284
|
-
<br>
|
|
1285
|
-
<br>
|
|
1286
|
-
<br>
|
|
1287
|
-
<br>
|
|
1288
|
-
<br>
|
|
1289
|
-
<br>
|
|
1290
|
-
<br>
|
|
1291
|
-
<br>
|
|
1292
|
-
|
|
1293
|
-
---
|
|
1294
|
-
|
|
1295
|
-
<br>
|
|
1296
|
-
<br>
|
|
1297
|
-
<br>
|
|
1298
|
-
<br>
|
|
1299
|
-
<br>
|
|
1300
|
-
<br>
|
|
1301
|
-
<br>
|
|
1302
|
-
<br>
|
|
1303
|
-
|
|
1304
|
-
## The full request lifecycle
|
|
1305
|
-
|
|
1306
|
-
```mermaid
|
|
1307
|
-
flowchart TB
|
|
1308
|
-
Client(["client"])
|
|
1309
|
-
Srv["Server thread<br/>accept_nonblock"]
|
|
1310
|
-
Rct["Reactor thread<br/>read_nonblock 64KB"]
|
|
1311
|
-
RP["Protocol-specific Ractor pool<br/>parses in own GVL"]
|
|
1312
|
-
Col["Collector threads<br/>drain Ractor::Ports"]
|
|
1313
|
-
ATP["App thread pool<br/>calls Rack app<br/>writes response"]
|
|
1314
|
-
|
|
1315
|
-
Client -->|"connected socket + request bytes"| Srv
|
|
1316
|
-
Srv -->|"fast path: complete on first read"| ATP
|
|
1317
|
-
Srv -->|"slow path: bytes not ready"| Rct
|
|
1318
|
-
Rct -->|"got bytes; hand raw buffer to pool"| RP
|
|
1319
|
-
RP --> Col
|
|
1320
|
-
Col -->|"complete"| ATP
|
|
1321
|
-
Col -->|"incomplete"| Rct
|
|
1322
|
-
ATP -->|"response bytes"| Client
|
|
1323
|
-
ATP -.->|"keep-alive: parse ready bytes inline"| ATP
|
|
1324
|
-
```
|
|
1325
|
-
|
|
1326
|
-
<br>
|
|
1327
|
-
<br>
|
|
1328
|
-
<br>
|
|
1329
|
-
<br>
|
|
1330
|
-
<br>
|
|
1331
|
-
<br>
|
|
1332
|
-
<br>
|
|
1333
|
-
<br>
|
|
1334
|
-
|
|
1335
|
-
---
|
|
1336
|
-
|
|
1337
|
-
<br>
|
|
1338
|
-
<br>
|
|
1339
|
-
<br>
|
|
1340
|
-
<br>
|
|
1341
|
-
<br>
|
|
1342
|
-
<br>
|
|
1343
|
-
<br>
|
|
1344
|
-
<br>
|
|
1345
|
-
|
|
1346
|
-
## Two paths through Raptor
|
|
1347
|
-
|
|
1348
|
-
**Fast path**
|
|
1349
|
-
|
|
1350
|
-
- <big>First `read_nonblock` gives us a complete request</big>
|
|
1351
|
-
- <big>Server thread (HTTP/1.1) or accepting pool worker (HTTP/2) parses inline</big>
|
|
1352
|
-
- <big>Pushes a proc to the app pool</big>
|
|
1353
|
-
- <big>No reactor. No Ractor pool.</big>
|
|
1354
|
-
- <big>Common for a normal request that fits in one TCP packet.</big>
|
|
1355
|
-
|
|
1356
|
-
**Slow path**
|
|
1357
|
-
|
|
1358
|
-
- <big>Bytes not ready, or the request is incomplete</big>
|
|
1359
|
-
- <big>Reactor takes over, waits for more bytes</big>
|
|
1360
|
-
- <big>When bytes arrive: read up to 64KB per syscall, hand the buffer to the protocol's Ractor pool</big>
|
|
1361
|
-
- 64KB is the per-read size, not a request size limit. Bigger requests just come in over more reads.
|
|
1362
|
-
- <big>Ractor parses on a **separate GVL**</big>
|
|
1363
|
-
- <big>Parsed env comes back via the collector, which pushes to the app pool</big>
|
|
1364
|
-
|
|
1365
|
-
<br>
|
|
1366
|
-
<br>
|
|
1367
|
-
<br>
|
|
1368
|
-
<br>
|
|
1369
|
-
<br>
|
|
1370
|
-
<br>
|
|
1371
|
-
<br>
|
|
1372
|
-
<br>
|
|
1373
|
-
|
|
1374
|
-
---
|
|
1375
|
-
|
|
1376
|
-
<br>
|
|
1377
|
-
<br>
|
|
1378
|
-
<br>
|
|
1379
|
-
<br>
|
|
1380
|
-
<br>
|
|
1381
|
-
<br>
|
|
1382
|
-
<br>
|
|
1383
|
-
<br>
|
|
1384
|
-
|
|
1385
|
-
## The eager keep-alive loop
|
|
1386
|
-
|
|
1387
|
-
After writing a response, if keep-alive is on:
|
|
1388
|
-
|
|
1389
|
-
```ruby
|
|
1390
|
-
loop do
|
|
1391
|
-
unless socket.wait_readable(0)
|
|
1392
|
-
reactor.persist(socket, id, ...)
|
|
1393
|
-
return
|
|
1394
|
-
end
|
|
1395
|
-
|
|
1396
|
-
# Bytes are ready. Parse the next request inline on this thread.
|
|
1397
|
-
# ...
|
|
1398
|
-
end
|
|
1399
|
-
```
|
|
1400
|
-
|
|
1401
|
-
- <big>Check for the next request on the same connection without waiting</big>
|
|
1402
|
-
- <big>If bytes are ready: parse and dispatch inline, on the same thread</big>
|
|
1403
|
-
- <big>Return to the reactor immediately when no bytes are ready, or when a request is incomplete</big>
|
|
1404
|
-
- <big>Back-to-back requests avoid a reactor round-trip without holding an app thread open</big>
|
|
1405
|
-
|
|
1406
|
-
<br>
|
|
1407
|
-
<br>
|
|
1408
|
-
<br>
|
|
1409
|
-
<br>
|
|
1410
|
-
<br>
|
|
1411
|
-
<br>
|
|
1412
|
-
<br>
|
|
1413
|
-
<br>
|
|
1414
|
-
|
|
1415
|
-
---
|
|
1416
|
-
|
|
1417
|
-
<br>
|
|
1418
|
-
<br>
|
|
1419
|
-
<br>
|
|
1420
|
-
<br>
|
|
1421
|
-
<br>
|
|
1422
|
-
<br>
|
|
1423
|
-
<br>
|
|
1424
|
-
<br>
|
|
1425
|
-
|
|
1426
|
-
## HTTP/2 in 60 seconds
|
|
1427
|
-
|
|
1428
|
-
For folks who mostly know HTTP/1.
|
|
1429
|
-
|
|
1430
|
-
**HTTP/1's limits**
|
|
1431
|
-
|
|
1432
|
-
- <big>One request at a time per connection</big>
|
|
1433
|
-
- <big>Head-of-line blocking (a slow response blocks the next one)</big>
|
|
1434
|
-
- <big>Text-based headers, repeated on every request, often kilobytes</big>
|
|
1435
|
-
|
|
1436
|
-
**HTTP/2 fixes**
|
|
1437
|
-
|
|
1438
|
-
- <big>**Binary framing**: the wire is small binary frames, not text</big>
|
|
1439
|
-
- <big>**Multiplexing**: many "streams" over one TCP connection, in parallel</big>
|
|
1440
|
-
- <big>**Header compression**: HPACK reuses previously-seen headers as small indexes</big>
|
|
1441
|
-
- <big>**Priorities and flow control**: server can send frames from many streams and negotiate windows</big>
|
|
1442
|
-
|
|
1443
|
-
One socket, many concurrent requests. Matters most when a client makes many small requests to the same server.
|
|
1444
|
-
|
|
1445
|
-
<br>
|
|
1446
|
-
<br>
|
|
1447
|
-
<br>
|
|
1448
|
-
<br>
|
|
1449
|
-
<br>
|
|
1450
|
-
<br>
|
|
1451
|
-
<br>
|
|
1452
|
-
<br>
|
|
1453
|
-
|
|
1454
|
-
---
|
|
1455
|
-
|
|
1456
|
-
<br>
|
|
1457
|
-
<br>
|
|
1458
|
-
<br>
|
|
1459
|
-
<br>
|
|
1460
|
-
<br>
|
|
1461
|
-
<br>
|
|
1462
|
-
<br>
|
|
1463
|
-
<br>
|
|
1464
|
-
|
|
1465
|
-
## HTTP/2 multiplexing
|
|
1466
|
-
|
|
1467
|
-
```mermaid
|
|
1468
|
-
flowchart LR
|
|
1469
|
-
C["Client"]
|
|
1470
|
-
S["Server"]
|
|
1471
|
-
|
|
1472
|
-
subgraph One TCP connection
|
|
1473
|
-
S1["Stream 1<br/>GET /users"]
|
|
1474
|
-
S2["Stream 3<br/>GET /posts"]
|
|
1475
|
-
S3["Stream 5<br/>POST /log"]
|
|
1476
|
-
S4["Stream 7<br/>GET /images/a.png"]
|
|
1477
|
-
end
|
|
1478
|
-
|
|
1479
|
-
C <-.->|"interleaved binary frames"| S
|
|
1480
|
-
S1 -.-> S
|
|
1481
|
-
S2 -.-> S
|
|
1482
|
-
S3 -.-> S
|
|
1483
|
-
S4 -.-> S
|
|
1484
|
-
```
|
|
1485
|
-
|
|
1486
|
-
- <big>Four requests in flight</big>
|
|
1487
|
-
- <big>One TCP connection</big>
|
|
1488
|
-
- <big>Frames from different streams are interleaved on the wire</big>
|
|
1489
|
-
- <big>Both sides reassemble by stream ID</big>
|
|
1490
|
-
|
|
1491
|
-
<br>
|
|
1492
|
-
<br>
|
|
1493
|
-
<br>
|
|
1494
|
-
<br>
|
|
1495
|
-
<br>
|
|
1496
|
-
<br>
|
|
1497
|
-
<br>
|
|
1498
|
-
<br>
|
|
1499
|
-
|
|
1500
|
-
---
|
|
1501
|
-
|
|
1502
|
-
<br>
|
|
1503
|
-
<br>
|
|
1504
|
-
<br>
|
|
1505
|
-
<br>
|
|
1506
|
-
<br>
|
|
1507
|
-
<br>
|
|
1508
|
-
<br>
|
|
1509
|
-
<br>
|
|
1510
|
-
|
|
1511
|
-
## HTTP/2 in Raptor
|
|
1512
|
-
|
|
1513
|
-
- <big>Native, on TLS connections</big>
|
|
1514
|
-
- <big>ALPN picks `h2` during the TLS handshake if the client offers it</big>
|
|
1515
|
-
- ALPN stands for Application-Layer Protocol Negotiation
|
|
1516
|
-
- It's a small step in the TLS handshake where client and server agree on which protocol to speak over the encrypted connection
|
|
1517
|
-
- <big>HPACK header compression, static Huffman table</big>
|
|
1518
|
-
- HPACK is HTTP/2's header compression scheme; it keeps a shared table of previously-seen headers so subsequent references are one small integer instead of the full header
|
|
1519
|
-
- <big>Own Ractor-safe C parser: `raptor_http2`</big>
|
|
1520
|
-
- <big>Puma [doesn't do HTTP/2](https://github.com/puma/puma/issues/2697), because they see it belonging at the edge (nginx, Caddy, ALB). Falcon does.</big>
|
|
1521
|
-
|
|
1522
|
-
Notable: a single HTTP/2 client connection in Raptor can have many streams in flight at once, spread across the app thread pool.
|
|
1523
|
-
|
|
1524
|
-
- <big>Each stream is a separate work item in the queue</big>
|
|
1525
|
-
- <big>Different streams from the same connection can end up on different app threads</big>
|
|
1526
|
-
- <big>Those threads still share the main GVL, so they overlap productively when the app is in I/O (the common Rails case), the same way Puma's keep-alive requests do</big>
|
|
1527
|
-
- <big>On the reactor path, frame batches parse in the HTTP/2 Ractor pool on its own GVL, in parallel with app work</big>
|
|
1528
|
-
|
|
1529
|
-
<br>
|
|
1530
|
-
<br>
|
|
1531
|
-
<br>
|
|
1532
|
-
<br>
|
|
1533
|
-
<br>
|
|
1534
|
-
<br>
|
|
1535
|
-
<br>
|
|
1536
|
-
<br>
|
|
1537
|
-
|
|
1538
|
-
---
|
|
1539
|
-
|
|
1540
|
-
<br>
|
|
1541
|
-
<br>
|
|
1542
|
-
<br>
|
|
1543
|
-
<br>
|
|
1544
|
-
<br>
|
|
1545
|
-
<br>
|
|
1546
|
-
<br>
|
|
1547
|
-
<br>
|
|
1548
|
-
|
|
1549
|
-
## The frame writer
|
|
1550
|
-
|
|
1551
|
-
- <big>One HTTP/2 connection carries many streams</big>
|
|
1552
|
-
- <big>Many app threads may be trying to write frames for their stream at once</big>
|
|
1553
|
-
- <big>Naïve: one mutex per connection. Contention grows with concurrent streams.</big>
|
|
1554
|
-
|
|
1555
|
-
Raptor stores the "pending frames" queue in an `Atom`:
|
|
1556
|
-
|
|
1557
|
-
- <big>Value is either `:idle` or an array of frames waiting to go out</big>
|
|
1558
|
-
- <big>A thread that wants to write does a CAS:</big>
|
|
1559
|
-
- If value is `:idle`: claim the writer, drain until empty
|
|
1560
|
-
- If value is an array: append your frames and return immediately. The current writer will pick them up.
|
|
1561
|
-
- <big>Only one thread does socket I/O at a time (a socket can only be written serially anyway)</big>
|
|
1562
|
-
- <big>But no thread ever blocks on a lock</big>
|
|
1563
|
-
|
|
1564
|
-
Same pattern for per-stream flow-control windows.
|
|
1565
|
-
|
|
1566
|
-
<br>
|
|
1567
|
-
<br>
|
|
1568
|
-
<br>
|
|
1569
|
-
<br>
|
|
1570
|
-
<br>
|
|
1571
|
-
<br>
|
|
1572
|
-
<br>
|
|
1573
|
-
<br>
|
|
1574
|
-
|
|
1575
|
-
---
|
|
1576
|
-
|
|
1577
|
-
<br>
|
|
1578
|
-
<br>
|
|
1579
|
-
<br>
|
|
1580
|
-
<br>
|
|
1581
|
-
<br>
|
|
1582
|
-
<br>
|
|
1583
|
-
<br>
|
|
1584
|
-
<br>
|
|
1585
|
-
|
|
1586
|
-
## Shared state
|
|
1587
|
-
|
|
1588
|
-
The master needs to know what the workers are doing.
|
|
1589
|
-
|
|
1590
|
-
- <big>Are they alive?</big>
|
|
1591
|
-
- <big>How busy?</big>
|
|
1592
|
-
- <big>Have they crashed?</big>
|
|
1593
|
-
|
|
1594
|
-
**Puma**: pipes. Workers write status messages every check interval. Master reads.
|
|
1595
|
-
|
|
1596
|
-
**Raptor**: anonymous shared memory via `mmap`.
|
|
1597
|
-
|
|
1598
|
-
<br>
|
|
1599
|
-
<br>
|
|
1600
|
-
<br>
|
|
1601
|
-
<br>
|
|
1602
|
-
<br>
|
|
1603
|
-
<br>
|
|
1604
|
-
<br>
|
|
1605
|
-
<br>
|
|
1606
|
-
|
|
1607
|
-
---
|
|
1608
|
-
|
|
1609
|
-
<br>
|
|
1610
|
-
<br>
|
|
1611
|
-
<br>
|
|
1612
|
-
<br>
|
|
1613
|
-
<br>
|
|
1614
|
-
<br>
|
|
1615
|
-
<br>
|
|
1616
|
-
<br>
|
|
1617
|
-
|
|
1618
|
-
## What is mmap?
|
|
1619
|
-
|
|
1620
|
-
- <big>`mmap(2)` is a system call</big>
|
|
1621
|
-
- <big>Asks the kernel for a region of virtual memory backed by something</big>
|
|
1622
|
-
- <big>Common uses:</big>
|
|
1623
|
-
- **File-backed**: mapping a file into memory (like Postgres's shared buffers)
|
|
1624
|
-
- **Anonymous** (`MAP_ANONYMOUS`): backed by nothing, just zero-filled pages
|
|
1625
|
-
- <big>The `MAP_SHARED` flag tells the kernel writes should be visible to other processes that also have the mapping</big>
|
|
1626
|
-
|
|
1627
|
-
<br>
|
|
1628
|
-
<br>
|
|
1629
|
-
<br>
|
|
1630
|
-
<br>
|
|
1631
|
-
<br>
|
|
1632
|
-
<br>
|
|
1633
|
-
<br>
|
|
1634
|
-
<br>
|
|
1635
|
-
|
|
1636
|
-
---
|
|
1637
|
-
|
|
1638
|
-
<br>
|
|
1639
|
-
<br>
|
|
1640
|
-
<br>
|
|
1641
|
-
<br>
|
|
1642
|
-
<br>
|
|
1643
|
-
<br>
|
|
1644
|
-
<br>
|
|
1645
|
-
<br>
|
|
1646
|
-
|
|
1647
|
-
## Anonymous shared memory
|
|
1648
|
-
|
|
1649
|
-
```mermaid
|
|
1650
|
-
flowchart TB
|
|
1651
|
-
subgraph Physical["Physical memory (kernel-managed)"]
|
|
1652
|
-
SHM[("shared pages")]
|
|
1653
|
-
end
|
|
1654
|
-
|
|
1655
|
-
subgraph P1["Process A"]
|
|
1656
|
-
A["Virtual address 0x7f...abc"]
|
|
1657
|
-
end
|
|
1658
|
-
|
|
1659
|
-
subgraph P2["Process B"]
|
|
1660
|
-
B["Virtual address 0x7f...def"]
|
|
1661
|
-
end
|
|
1662
|
-
|
|
1663
|
-
A -.->|"maps to"| SHM
|
|
1664
|
-
B -.->|"maps to"| SHM
|
|
1665
|
-
```
|
|
1666
|
-
|
|
1667
|
-
- <big>Two processes call `mmap(MAP_ANONYMOUS | MAP_SHARED)` on the same region</big>
|
|
1668
|
-
- <big>Or, more usefully: **one process calls `mmap`, then forks**</big>
|
|
1669
|
-
- <big>Both processes end up with virtual addresses that point at the same physical pages</big>
|
|
1670
|
-
- <big>A write in process A is visible in process B, immediately</big>
|
|
1671
|
-
- <big>No syscalls to synchronise. It's just memory.</big>
|
|
1672
|
-
|
|
1673
|
-
<br>
|
|
1674
|
-
<br>
|
|
1675
|
-
<br>
|
|
1676
|
-
<br>
|
|
1677
|
-
<br>
|
|
1678
|
-
<br>
|
|
1679
|
-
<br>
|
|
1680
|
-
<br>
|
|
1681
|
-
|
|
1682
|
-
---
|
|
1683
|
-
|
|
1684
|
-
<br>
|
|
1685
|
-
<br>
|
|
1686
|
-
<br>
|
|
1687
|
-
<br>
|
|
1688
|
-
<br>
|
|
1689
|
-
<br>
|
|
1690
|
-
<br>
|
|
1691
|
-
<br>
|
|
1692
|
-
|
|
1693
|
-
## Raptor's stats via mmap
|
|
1694
|
-
|
|
1695
|
-
```mermaid
|
|
1696
|
-
flowchart TB
|
|
1697
|
-
subgraph Kernel["Kernel-managed shared pages"]
|
|
1698
|
-
SHM[("stats region")]
|
|
1699
|
-
end
|
|
1700
|
-
|
|
1701
|
-
M["Master"] <-.->|"reads directly"| SHM
|
|
1702
|
-
W1["Worker 0"] -->|"writes 49 bytes"| SHM
|
|
1703
|
-
W2["Worker 1"] -->|"writes 49 bytes"| SHM
|
|
1704
|
-
WN["Worker N"] -->|"writes 49 bytes"| SHM
|
|
1705
|
-
```
|
|
1706
|
-
|
|
1707
|
-
- <big>Master calls `mmap` for the region **before** forking</big>
|
|
1708
|
-
- <big>Every worker inherits the mapping</big>
|
|
1709
|
-
- <big>Each worker writes a 49-byte slot every second: pid, phase, requests, backlog, busy and available threads, boot time, checkin time, booted flag</big>
|
|
1710
|
-
- <big>Master reads the whole region directly. No JSON. No pipe drain. No signal.</big>
|
|
1711
|
-
- <big>`bundle exec raptor stats` prints the region as JSON, essentially instantly</big>
|
|
1712
|
-
- <big>An optional read-only Unix socket exposes the cluster snapshot at `GET /stats` for monitoring</big>
|
|
1713
|
-
|
|
1714
|
-
Wrapped in a small C extension I wrote: **`mmap-ruby`**.
|
|
1715
|
-
|
|
1716
|
-
<br>
|
|
1717
|
-
<br>
|
|
1718
|
-
<br>
|
|
1719
|
-
<br>
|
|
1720
|
-
<br>
|
|
1721
|
-
<br>
|
|
1722
|
-
<br>
|
|
1723
|
-
<br>
|
|
1724
|
-
|
|
1725
|
-
---
|
|
1726
|
-
|
|
1727
|
-
<br>
|
|
1728
|
-
<br>
|
|
1729
|
-
<br>
|
|
1730
|
-
<br>
|
|
1731
|
-
<br>
|
|
1732
|
-
<br>
|
|
1733
|
-
<br>
|
|
1734
|
-
<br>
|
|
1735
|
-
|
|
1736
|
-
## But how do the writes not clash?
|
|
1737
|
-
|
|
1738
|
-
Fair question. This is shared memory. What stops two workers from stepping on each other?
|
|
1739
|
-
|
|
1740
|
-
**The layout does the work**
|
|
1741
|
-
|
|
1742
|
-
- <big>Each worker owns exactly one slot in the region</big>
|
|
1743
|
-
- Slot address = base + (worker index × 49 bytes)
|
|
1744
|
-
- <big>Only that one worker ever writes to that slot</big>
|
|
1745
|
-
- <big>The master only ever reads. It never writes.</big>
|
|
1746
|
-
- <big>No two writers touch the same bytes, so there is no write-write contention</big>
|
|
1747
|
-
|
|
1748
|
-
**What about a read happening mid-write?**
|
|
1749
|
-
|
|
1750
|
-
- <big>The master can see a slot half-updated: a "torn read"</big>
|
|
1751
|
-
- <big>For stats, that's fine</big>
|
|
1752
|
-
- <big>These are eventually-consistent metrics, not authoritative state</big>
|
|
1753
|
-
- <big>The next second's checkin makes it right</big>
|
|
1754
|
-
|
|
1755
|
-
**If we did need atomicity**
|
|
1756
|
-
|
|
1757
|
-
- <big>We'd use atomic word writes (like the BPF load map does)</big>
|
|
1758
|
-
- <big>Or a per-slot version counter the reader could double-check</big>
|
|
1759
|
-
- <big>Neither is worth the complexity for stats</big>
|
|
1760
|
-
|
|
1761
|
-
<br>
|
|
1762
|
-
<br>
|
|
1763
|
-
<br>
|
|
1764
|
-
<br>
|
|
1765
|
-
<br>
|
|
1766
|
-
<br>
|
|
1767
|
-
<br>
|
|
1768
|
-
<br>
|
|
1769
|
-
|
|
1770
|
-
---
|
|
1771
|
-
|
|
1772
|
-
<br>
|
|
1773
|
-
<br>
|
|
1774
|
-
<br>
|
|
1775
|
-
<br>
|
|
1776
|
-
<br>
|
|
1777
|
-
<br>
|
|
1778
|
-
<br>
|
|
1779
|
-
<br>
|
|
1780
|
-
|
|
1781
|
-
## Load-aware routing
|
|
1782
|
-
|
|
1783
|
-
This is my favourite piece.
|
|
1784
|
-
|
|
1785
|
-
The problem:
|
|
1786
|
-
|
|
1787
|
-
- <big>N workers all bind the same TCP port with `SO_REUSEPORT`</big>
|
|
1788
|
-
- `SO_REUSEPORT` is a socket option that lets multiple processes bind the same port at the same time
|
|
1789
|
-
- The kernel treats them as a group and hands out incoming connections between them
|
|
1790
|
-
- <big>Kernel picks which listener gets each incoming SYN</big>
|
|
1791
|
-
- "SYN" is the first packet of a new TCP connection
|
|
1792
|
-
- <big>Default is a deterministic hash of the connection's four-tuple (client IP, client port, server IP, server port)</big>
|
|
1793
|
-
- Same four-tuple always lands on the same worker. Not random. Not round-robin.
|
|
1794
|
-
- Good spread on average across many clients, but no awareness of which worker is actually busy
|
|
1795
|
-
|
|
1796
|
-
The Linux answer since kernel 4.5: **attach a BPF program to the reuseport group** via the `SO_ATTACH_REUSEPORT_EBPF` socket option. When the kernel selects a socket from that group for a new connection, your program can make the choice.
|
|
1797
|
-
|
|
1798
|
-
Raptor uses this for plain `tcp://` listeners. TLS and Unix listeners continue to use sockets inherited from the master.
|
|
1799
|
-
|
|
1800
|
-
<br>
|
|
1801
|
-
<br>
|
|
1802
|
-
<br>
|
|
1803
|
-
<br>
|
|
1804
|
-
<br>
|
|
1805
|
-
<br>
|
|
1806
|
-
<br>
|
|
1807
|
-
<br>
|
|
1808
|
-
|
|
1809
|
-
---
|
|
1810
|
-
|
|
1811
|
-
<br>
|
|
1812
|
-
<br>
|
|
1813
|
-
<br>
|
|
1814
|
-
<br>
|
|
1815
|
-
<br>
|
|
1816
|
-
<br>
|
|
1817
|
-
<br>
|
|
1818
|
-
<br>
|
|
1819
|
-
|
|
1820
|
-
## What Puma does instead
|
|
1821
|
-
|
|
1822
|
-
Puma has [`ClusterAcceptLoopDelay`](https://github.com/puma/puma/blob/master/lib/puma/cluster_accept_loop_delay.rb) since Puma 7.
|
|
1823
|
-
|
|
1824
|
-
- <big>Before each accept, a worker sleeps for a fraction of `max_delay` (default 5ms)</big>
|
|
1825
|
-
- <big>The sleep is proportional to the worker's own load: 0 when idle, `max_delay` when very busy</big>
|
|
1826
|
-
- <big>Every worker still accepts. Nobody refuses. Nobody gets skipped.</big>
|
|
1827
|
-
- <big>Busier workers wake later. Less-busy workers wake first and win the accept race on the listener inherited from Puma's master.</big>
|
|
1828
|
-
|
|
1829
|
-
It's clever, portable, and deliberately imprecise. Each worker only needs its own load. Raptor's BPF path instead publishes cluster-wide load to a kernel map and chooses before a worker calls `accept`.
|
|
1830
|
-
|
|
1831
|
-
<br>
|
|
1832
|
-
<br>
|
|
1833
|
-
<br>
|
|
1834
|
-
<br>
|
|
1835
|
-
<br>
|
|
1836
|
-
<br>
|
|
1837
|
-
<br>
|
|
1838
|
-
<br>
|
|
1839
|
-
|
|
1840
|
-
---
|
|
1841
|
-
|
|
1842
|
-
<br>
|
|
1843
|
-
<br>
|
|
1844
|
-
<br>
|
|
1845
|
-
<br>
|
|
1846
|
-
<br>
|
|
1847
|
-
<br>
|
|
1848
|
-
<br>
|
|
1849
|
-
<br>
|
|
1850
|
-
|
|
1851
|
-
## What is BPF?
|
|
1852
|
-
|
|
1853
|
-
- <big>BPF stands for Berkeley Packet Filter (historical name from packet capture)</big>
|
|
1854
|
-
- <big>Now it's a small virtual machine inside the Linux kernel</big>
|
|
1855
|
-
- <big>Programs written in a restricted subset of C</big>
|
|
1856
|
-
- <big>Compiled with `clang -target bpf` to BPF bytecode</big>
|
|
1857
|
-
- <big>Kernel **verifies** the program before letting it run:</big>
|
|
1858
|
-
- Bounded loops
|
|
1859
|
-
- No out-of-bounds memory access
|
|
1860
|
-
- Provably terminates
|
|
1861
|
-
- <big>If any of that can't be proved, the program is rejected at load time</big>
|
|
1862
|
-
- <big>Once loaded, runs inside the kernel without a userspace round-trip for each decision</big>
|
|
1863
|
-
|
|
1864
|
-
Modern Linux observability (`bcc`, `bpftrace`, `cilium`, `perf`) is all built on this. It's genuinely one of the most exciting things that has happened to Linux in the last decade.
|
|
1865
|
-
|
|
1866
|
-
<br>
|
|
1867
|
-
<br>
|
|
1868
|
-
<br>
|
|
1869
|
-
<br>
|
|
1870
|
-
<br>
|
|
1871
|
-
<br>
|
|
1872
|
-
<br>
|
|
1873
|
-
<br>
|
|
1874
|
-
|
|
1875
|
-
---
|
|
1876
|
-
|
|
1877
|
-
<br>
|
|
1878
|
-
<br>
|
|
1879
|
-
<br>
|
|
1880
|
-
<br>
|
|
1881
|
-
<br>
|
|
1882
|
-
<br>
|
|
1883
|
-
<br>
|
|
1884
|
-
<br>
|
|
1885
|
-
|
|
1886
|
-
## Raptor's BPF program
|
|
1887
|
-
|
|
1888
|
-
```mermaid
|
|
1889
|
-
flowchart TB
|
|
1890
|
-
Client(["new connection"])
|
|
1891
|
-
Hash["connection hash"]
|
|
1892
|
-
A["candidate A<br/>Worker 0, load 5"]
|
|
1893
|
-
B["candidate B<br/>Worker 3, load 6"]
|
|
1894
|
-
Pick["choose Worker 0<br/>reserve load 5 → 6"]
|
|
1895
|
-
Queue["Worker 0 accept queue"]
|
|
1896
|
-
|
|
1897
|
-
Client --> Hash
|
|
1898
|
-
Hash -->|"sample 1"| A
|
|
1899
|
-
Hash -->|"sample 2"| B
|
|
1900
|
-
A --> Pick
|
|
1901
|
-
B --> Pick
|
|
1902
|
-
Pick -->|"atomic increment, then route"| Queue
|
|
1903
|
-
```
|
|
1904
|
-
|
|
1905
|
-
- <big>Each worker has a **load reporter thread**</big>
|
|
1906
|
-
- <big>Publishes its current backlog into a BPF map every millisecond</big>
|
|
1907
|
-
- <big>The connection hash picks two distinct workers</big>
|
|
1908
|
-
- <big>The BPF program compares those two load slots and chooses the lower one</big>
|
|
1909
|
-
- <big>That is **power of two choices**: near-global balance without scanning every worker</big>
|
|
1910
|
-
|
|
1911
|
-
<br>
|
|
1912
|
-
<br>
|
|
1913
|
-
<br>
|
|
1914
|
-
<br>
|
|
1915
|
-
<br>
|
|
1916
|
-
<br>
|
|
1917
|
-
<br>
|
|
1918
|
-
<br>
|
|
1919
|
-
|
|
1920
|
-
---
|
|
1921
|
-
|
|
1922
|
-
<br>
|
|
1923
|
-
<br>
|
|
1924
|
-
<br>
|
|
1925
|
-
<br>
|
|
1926
|
-
<br>
|
|
1927
|
-
<br>
|
|
1928
|
-
<br>
|
|
1929
|
-
<br>
|
|
1930
|
-
|
|
1931
|
-
## Why the reservation matters
|
|
1932
|
-
|
|
1933
|
-
The reporter only publishes once per millisecond. A burst can contain many connections.
|
|
1934
|
-
|
|
1935
|
-
- <big>Without a reservation, every connection in that burst can observe the same stale low value</big>
|
|
1936
|
-
- <big>They can all choose the same worker before Ruby reports the new backlog</big>
|
|
1937
|
-
- <big>Raptor atomically increments the chosen BPF slot **before** routing</big>
|
|
1938
|
-
- <big>The next connection sees that reservation immediately</big>
|
|
1939
|
-
- <big>When Ruby accepts a socket, it also publishes `backlog + 1` rather than waiting for the reporter</big>
|
|
1940
|
-
|
|
1941
|
-
The load number is partly measurement and partly admission ledger. That closes the stale-reporting window that caused herding in the earlier design.
|
|
1942
|
-
|
|
1943
|
-
<br>
|
|
1944
|
-
<br>
|
|
1945
|
-
<br>
|
|
1946
|
-
<br>
|
|
1947
|
-
<br>
|
|
1948
|
-
<br>
|
|
1949
|
-
<br>
|
|
1950
|
-
<br>
|
|
1951
|
-
|
|
1952
|
-
---
|
|
1953
|
-
|
|
1954
|
-
<br>
|
|
1955
|
-
<br>
|
|
1956
|
-
<br>
|
|
1957
|
-
<br>
|
|
1958
|
-
<br>
|
|
1959
|
-
<br>
|
|
1960
|
-
<br>
|
|
1961
|
-
<br>
|
|
1962
|
-
|
|
1963
|
-
## BPF in Raptor, by the numbers
|
|
1964
|
-
|
|
1965
|
-
- <big>The whole BPF program is under 70 lines of C</big>
|
|
1966
|
-
- <big>Compiles down to a few hundred bytes of BPF bytecode</big>
|
|
1967
|
-
- <big>Runs in the kernel whenever the reuseport group selects a socket for a new connection</big>
|
|
1968
|
-
- <big>To load it from Ruby, I wrote **`libbpf-ruby`**</big>
|
|
1969
|
-
- `libbpf` is the standard C library that user-space programs use to load, verify and attach BPF programs
|
|
1970
|
-
- `libbpf-ruby` is my binding around it
|
|
1971
|
-
- <big>As far as I know, it's the only Ruby binding for libbpf</big>
|
|
1972
|
-
|
|
1973
|
-
Graceful fallback:
|
|
1974
|
-
|
|
1975
|
-
- <big>If `libbpf-ruby` isn't installed, or the BPF object hasn't been compiled, Raptor silently falls back to the listener inherited from the master</big>
|
|
1976
|
-
- <big>Everything still works. You just don't get load-aware routing.</big>
|
|
1977
|
-
- <big>If the kernel refuses the program (verifier error, missing features), startup raises. Loud failure, not silent misbehaviour.</big>
|
|
1978
|
-
|
|
1979
|
-
<br>
|
|
1980
|
-
<br>
|
|
1981
|
-
<br>
|
|
1982
|
-
<br>
|
|
1983
|
-
<br>
|
|
1984
|
-
<br>
|
|
1985
|
-
<br>
|
|
1986
|
-
<br>
|
|
1987
|
-
|
|
1988
|
-
---
|
|
1989
|
-
|
|
1990
|
-
<br>
|
|
1991
|
-
<br>
|
|
1992
|
-
<br>
|
|
1993
|
-
<br>
|
|
1994
|
-
<br>
|
|
1995
|
-
<br>
|
|
1996
|
-
<br>
|
|
1997
|
-
<br>
|
|
1998
|
-
|
|
1999
|
-
## Zooming out: the master process
|
|
2000
|
-
|
|
2001
|
-
We've walked every worker-side thing Raptor does. mmap and BPF are two of the pieces the master itself owns and hands to workers at boot. Let me close out with the rest of that master-side story.
|
|
2002
|
-
|
|
2003
|
-
<br>
|
|
2004
|
-
<br>
|
|
2005
|
-
<br>
|
|
2006
|
-
<br>
|
|
2007
|
-
<br>
|
|
2008
|
-
<br>
|
|
2009
|
-
<br>
|
|
2010
|
-
<br>
|
|
2011
|
-
|
|
2012
|
-
---
|
|
2013
|
-
|
|
2014
|
-
<br>
|
|
2015
|
-
<br>
|
|
2016
|
-
<br>
|
|
2017
|
-
<br>
|
|
2018
|
-
<br>
|
|
2019
|
-
<br>
|
|
2020
|
-
<br>
|
|
2021
|
-
<br>
|
|
2022
|
-
|
|
2023
|
-
## Preforking
|
|
2024
|
-
|
|
2025
|
-
Every clustered Ruby web server does this pattern:
|
|
2026
|
-
|
|
2027
|
-
- <big>Master process boots</big>
|
|
2028
|
-
- <big>Master loads your Rack app</big>
|
|
2029
|
-
- <big>Master forks N worker processes</big>
|
|
2030
|
-
- <big>Workers inherit the loaded app</big>
|
|
2031
|
-
|
|
2032
|
-
Why load in the master? Copy-on-write.
|
|
2033
|
-
|
|
2034
|
-
**In Raptor, this is always on**
|
|
2035
|
-
|
|
2036
|
-
- <big>No knob to turn it off</big>
|
|
2037
|
-
- <big>Your Rack app has to be fork-safe: no threads at load time, no file handles that need to survive the fork, no global connections opened during `require`</big>
|
|
2038
|
-
- <big>Puma had this as opt-in until v7. Raptor's opinionated on it.</big>
|
|
2039
|
-
|
|
2040
|
-
<br>
|
|
2041
|
-
<br>
|
|
2042
|
-
<br>
|
|
2043
|
-
<br>
|
|
2044
|
-
<br>
|
|
2045
|
-
<br>
|
|
2046
|
-
<br>
|
|
2047
|
-
<br>
|
|
2048
|
-
|
|
2049
|
-
---
|
|
2050
|
-
|
|
2051
|
-
<br>
|
|
2052
|
-
<br>
|
|
2053
|
-
<br>
|
|
2054
|
-
<br>
|
|
2055
|
-
<br>
|
|
2056
|
-
<br>
|
|
2057
|
-
<br>
|
|
2058
|
-
<br>
|
|
2059
|
-
|
|
2060
|
-
## Fork and copy-on-write
|
|
2061
|
-
|
|
2062
|
-
```mermaid
|
|
2063
|
-
flowchart TB
|
|
2064
|
-
subgraph Master["Master (before fork)"]
|
|
2065
|
-
MP["Loaded Rack app<br/>~100MB of pages"]
|
|
2066
|
-
end
|
|
2067
|
-
|
|
2068
|
-
subgraph After["After fork()"]
|
|
2069
|
-
subgraph W0["Worker 0"]
|
|
2070
|
-
W0V["virtual memory"]
|
|
2071
|
-
end
|
|
2072
|
-
subgraph W1["Worker 1"]
|
|
2073
|
-
W1V["virtual memory"]
|
|
2074
|
-
end
|
|
2075
|
-
subgraph MS["Master"]
|
|
2076
|
-
MSV["virtual memory"]
|
|
2077
|
-
end
|
|
2078
|
-
|
|
2079
|
-
SP[("Shared physical pages")]
|
|
2080
|
-
|
|
2081
|
-
W0V -.->|"read-only pointer"| SP
|
|
2082
|
-
W1V -.->|"read-only pointer"| SP
|
|
2083
|
-
MSV -.->|"read-only pointer"| SP
|
|
2084
|
-
end
|
|
2085
|
-
|
|
2086
|
-
Master ==> After
|
|
2087
|
-
```
|
|
2088
|
-
|
|
2089
|
-
- <big>All three processes point at the same physical memory</big>
|
|
2090
|
-
- <big>Nobody is using 100MB each. They share the 100MB.</big>
|
|
2091
|
-
- <big>The moment any of them writes to a page, the kernel copies just that page for the writer</big>
|
|
2092
|
-
- <big>So the shared memory stays shared until someone dirties it</big>
|
|
2093
|
-
|
|
2094
|
-
On a real Rails app, this saves gigabytes at startup.
|
|
2095
|
-
|
|
2096
|
-
<br>
|
|
2097
|
-
<br>
|
|
2098
|
-
<br>
|
|
2099
|
-
<br>
|
|
2100
|
-
<br>
|
|
2101
|
-
<br>
|
|
2102
|
-
<br>
|
|
2103
|
-
<br>
|
|
2104
|
-
|
|
2105
|
-
---
|
|
2106
|
-
|
|
2107
|
-
<br>
|
|
2108
|
-
<br>
|
|
2109
|
-
<br>
|
|
2110
|
-
<br>
|
|
2111
|
-
<br>
|
|
2112
|
-
<br>
|
|
2113
|
-
<br>
|
|
2114
|
-
<br>
|
|
2115
|
-
|
|
2116
|
-
## Refork
|
|
2117
|
-
|
|
2118
|
-
- <big>Copy-on-write savings decay over time</big>
|
|
2119
|
-
- <big>Workers dirty pages as they serve traffic</big>
|
|
2120
|
-
- <big>Allocations, YJIT-compiled blocks (YJIT is Ruby's in-process JIT compiler), gem lazy-loads, cache warm-up</big>
|
|
2121
|
-
- <big>After a few hours, each worker's memory has climbed toward its full RSS</big>
|
|
2122
|
-
- RSS is "resident set size", the amount of physical memory the process is actually using
|
|
2123
|
-
- <big>The saving from preloading is mostly gone</big>
|
|
2124
|
-
|
|
2125
|
-
The classic solution: **refork off a worker that's been serving traffic**, not off the master.
|
|
2126
|
-
|
|
2127
|
-
```mermaid
|
|
2128
|
-
sequenceDiagram
|
|
2129
|
-
participant M as Master
|
|
2130
|
-
participant W as Warm worker
|
|
2131
|
-
participant S as Seed (was W)
|
|
2132
|
-
participant N as New worker
|
|
2133
|
-
|
|
2134
|
-
Note over M,N: Every `refork_after` requests
|
|
2135
|
-
M->>W: SIGURG
|
|
2136
|
-
W->>W: Drain in-flight requests
|
|
2137
|
-
W->>M: "I'm the seed now"
|
|
2138
|
-
Note over S: Was serving. Now pure fork source.
|
|
2139
|
-
M->>S: "Fork worker N"
|
|
2140
|
-
S->>N: fork()
|
|
2141
|
-
N->>M: reparented via PR_SET_CHILD_SUBREAPER
|
|
2142
|
-
Note over M,N: New worker inherits the warm VM
|
|
2143
|
-
```
|
|
2144
|
-
|
|
2145
|
-
- <big>`SIGURG` is one of the Unix signals a process can trap, chosen here because nothing else uses it</big>
|
|
2146
|
-
- <big>The pattern was invented by Shopify's [Pitchfork](https://github.com/Shopify/pitchfork)</big>
|
|
2147
|
-
- <big>Puma is catching up to the same idea via Instacart's [`mold_worker` PR](https://github.com/puma/puma/pull/3643)</big>
|
|
2148
|
-
- <big>Raptor's version is native, controlled by a `refork_after` config option</big>
|
|
2149
|
-
|
|
2150
|
-
<br>
|
|
2151
|
-
<br>
|
|
2152
|
-
<br>
|
|
2153
|
-
<br>
|
|
2154
|
-
<br>
|
|
2155
|
-
<br>
|
|
2156
|
-
<br>
|
|
2157
|
-
<br>
|
|
2158
|
-
|
|
2159
|
-
---
|
|
2160
|
-
|
|
2161
|
-
<br>
|
|
2162
|
-
<br>
|
|
2163
|
-
<br>
|
|
2164
|
-
<br>
|
|
2165
|
-
<br>
|
|
2166
|
-
<br>
|
|
2167
|
-
<br>
|
|
2168
|
-
<br>
|
|
2169
|
-
|
|
2170
|
-
## The subreaper trick
|
|
2171
|
-
|
|
2172
|
-
- <big>Normally, a process's children reparent to `init` (PID 1) when the direct parent dies</big>
|
|
2173
|
-
- `init` is the very first process the kernel starts; its job is to "adopt" orphaned processes
|
|
2174
|
-
- <big>Linux has a syscall: `prctl(PR_SET_CHILD_SUBREAPER)`</big>
|
|
2175
|
-
- A syscall is a request from a program to the kernel to do something the program can't do itself
|
|
2176
|
-
- <big>Marks a process as an "alternate reaper"</big>
|
|
2177
|
-
- <big>Grandchildren reparent to *it* instead of `init`</big>
|
|
2178
|
-
- <big>That's how the master keeps tracking workers forked by a seed, not directly</big>
|
|
2179
|
-
|
|
2180
|
-
Wrapped in a tiny C extension I wrote called `raptor_native`.
|
|
2181
|
-
|
|
2182
|
-
Linux-only. On macOS, `refork_after` is ignored with a warning at boot.
|
|
2183
|
-
|
|
2184
|
-
<br>
|
|
2185
|
-
<br>
|
|
2186
|
-
<br>
|
|
2187
|
-
<br>
|
|
2188
|
-
<br>
|
|
2189
|
-
<br>
|
|
2190
|
-
<br>
|
|
2191
|
-
<br>
|
|
2192
|
-
|
|
2193
|
-
---
|
|
2194
|
-
|
|
2195
|
-
<br>
|
|
2196
|
-
<br>
|
|
2197
|
-
<br>
|
|
2198
|
-
<br>
|
|
2199
|
-
<br>
|
|
2200
|
-
<br>
|
|
2201
|
-
<br>
|
|
2202
|
-
<br>
|
|
2203
|
-
|
|
2204
|
-
## The gem-shed
|
|
2205
|
-
|
|
2206
|
-
The libraries that came out of building Raptor:
|
|
2207
|
-
|
|
2208
|
-
- <big>**`atomic-ruby`** – CAS-based primitives (`Atom`, `AtomicThreadPool`, ...)</big>
|
|
2209
|
-
- <big>**`ractor-pool`** – coordinator-dispatch pool for Ractor-based pipelines</big>
|
|
2210
|
-
- <big>**`red-black-tree`** – self-balancing binary search tree</big>
|
|
2211
|
-
- <big>**`mmap-ruby`** – `mmap(2)` binding for cross-process shared memory</big>
|
|
2212
|
-
- <big>**`libbpf-ruby`** – `libbpf` binding, load and attach BPF programs from Ruby</big>
|
|
2213
|
-
|
|
2214
|
-
Each of them is small, focused, tested, and useful outside of Raptor.
|
|
2215
|
-
|
|
2216
|
-
- <big>If you want lock-free primitives in Ruby, `atomic-ruby` gives you a small, direct API</big>
|
|
2217
|
-
- <big>If you want to try Ractor-based parallelism without writing the coordination yourself, `ractor-pool` is that</big>
|
|
2218
|
-
- <big>If you want to load a BPF program from Ruby, `libbpf-ruby` gives you a direct binding to libbpf</big>
|
|
2219
|
-
|
|
2220
|
-
I didn't set out to build a small library ecosystem. It's what happens when you refuse to fold every helper back into the main gem.
|
|
2221
|
-
|
|
2222
|
-
<br>
|
|
2223
|
-
<br>
|
|
2224
|
-
<br>
|
|
2225
|
-
<br>
|
|
2226
|
-
<br>
|
|
2227
|
-
<br>
|
|
2228
|
-
<br>
|
|
2229
|
-
<br>
|
|
2230
|
-
|
|
2231
|
-
---
|
|
2232
|
-
|
|
2233
|
-
<br>
|
|
2234
|
-
<br>
|
|
2235
|
-
<br>
|
|
2236
|
-
<br>
|
|
2237
|
-
<br>
|
|
2238
|
-
<br>
|
|
2239
|
-
<br>
|
|
2240
|
-
<br>
|
|
2241
|
-
|
|
2242
|
-
## The numbers, in shape
|
|
2243
|
-
|
|
2244
|
-
Real numbers are in the [README benchmarks section](../README.md#micro-benchmarks). The shape:
|
|
2245
|
-
|
|
2246
|
-
- <big>**IO-bound HTTP/1.1**: Scaling lifts Raptor from 3.11k to 8.63k req/s without keep-alive and from 3.21k to 9.43k with it. Falcon wins the first; Raptor wins the second.</big>
|
|
2247
|
-
- <big>**CPU-bound HTTP/1.1**: Fixed and scaling Raptor are effectively identical. Puma leads by 5% without keep-alive and less than 1% with it; Raptor leads Falcon.</big>
|
|
2248
|
-
- Tail latency ("p95") is the response time that 5% of requests exceed. It's what your slowest users see. Lower is better.
|
|
2249
|
-
- <big>**HTTP/2**: Scaling lifts Raptor from 1.14k to 4.46k req/s on IO and narrows Falcon's CPU-throughput lead from 27% to 14%. Falcon still leads throughput; Raptor has the lower CPU p95.</big>
|
|
2250
|
-
- <big>**Variance**: HTTP/1.1 is stable. HTTP/2 is noisy enough that I treat it as direction, not a precise ranking.</big>
|
|
2251
|
-
|
|
2252
|
-
Different workloads, different winners. That's fine.
|
|
2253
|
-
|
|
2254
|
-
<br>
|
|
2255
|
-
<br>
|
|
2256
|
-
<br>
|
|
2257
|
-
<br>
|
|
2258
|
-
<br>
|
|
2259
|
-
<br>
|
|
2260
|
-
<br>
|
|
2261
|
-
<br>
|
|
2262
|
-
|
|
2263
|
-
---
|
|
2264
|
-
|
|
2265
|
-
<br>
|
|
2266
|
-
<br>
|
|
2267
|
-
<br>
|
|
2268
|
-
<br>
|
|
2269
|
-
<br>
|
|
2270
|
-
<br>
|
|
2271
|
-
<br>
|
|
2272
|
-
<br>
|
|
2273
|
-
|
|
2274
|
-
## What you give up
|
|
2275
|
-
|
|
2276
|
-
**Battle-tested**
|
|
2277
|
-
|
|
2278
|
-
- <big>Puma has been in production Rails since 2011</big>
|
|
2279
|
-
- <big>Raptor has been running my benchmarks since 2026</big>
|
|
2280
|
-
- <big>Those are not the same thing.</big>
|
|
2281
|
-
|
|
2282
|
-
**Ruby version**
|
|
2283
|
-
|
|
2284
|
-
- <big>Raptor needs Ruby 4.0</big>
|
|
2285
|
-
- <big>`Ractor::Port` and Ractor internals only just stabilised</big>
|
|
2286
|
-
- <big>If you're on 3.x, Raptor isn't for you yet</big>
|
|
2287
|
-
|
|
2288
|
-
**Debugging is harder**
|
|
2289
|
-
|
|
2290
|
-
- <big>Control flow crosses Ractor and thread boundaries</big>
|
|
2291
|
-
- <big>Tracing a request end-to-end means stitching several stack traces together</big>
|
|
2292
|
-
- <big>Real cost of the architecture, not a temporary bug</big>
|
|
2293
|
-
|
|
2294
|
-
<br>
|
|
2295
|
-
<br>
|
|
2296
|
-
<br>
|
|
2297
|
-
<br>
|
|
2298
|
-
<br>
|
|
2299
|
-
<br>
|
|
2300
|
-
<br>
|
|
2301
|
-
<br>
|
|
2302
|
-
|
|
2303
|
-
---
|
|
2304
|
-
|
|
2305
|
-
<br>
|
|
2306
|
-
<br>
|
|
2307
|
-
<br>
|
|
2308
|
-
<br>
|
|
2309
|
-
<br>
|
|
2310
|
-
<br>
|
|
2311
|
-
<br>
|
|
2312
|
-
<br>
|
|
2313
|
-
|
|
2314
|
-
## Why you should try it
|
|
2315
|
-
|
|
2316
|
-
I'm not asking you to swap your production Rails deploy tomorrow. I'm asking you to be curious.
|
|
2317
|
-
|
|
2318
|
-
- <big>Ruby 4 landed</big>
|
|
2319
|
-
- <big>Ractors are stable now</big>
|
|
2320
|
-
- <big>`Ractor::Port` is stable</big>
|
|
2321
|
-
- <big>We can compile BPF programs and load them from Ruby</big>
|
|
2322
|
-
- <big>We can `mmap(MAP_ANONYMOUS)` for cross-process shared memory</big>
|
|
2323
|
-
- <big>We can build lock-free data structures via CAS in Ruby</big>
|
|
2324
|
-
- <big>We can wire syscalls like `prctl` and `sched_setaffinity` into native extensions in a few dozen lines of C</big>
|
|
2325
|
-
|
|
2326
|
-
Every one of these was possible individually before. What's new is that it's now practical to combine them.
|
|
2327
|
-
|
|
2328
|
-
Raptor is what happens when you take that seriously and build a web server that isn't shaped by the constraints of Ruby 2.
|
|
2329
|
-
|
|
2330
|
-
<br>
|
|
2331
|
-
<br>
|
|
2332
|
-
<br>
|
|
2333
|
-
<br>
|
|
2334
|
-
<br>
|
|
2335
|
-
<br>
|
|
2336
|
-
<br>
|
|
2337
|
-
<br>
|
|
2338
|
-
|
|
2339
|
-
---
|
|
2340
|
-
|
|
2341
|
-
<br>
|
|
2342
|
-
<br>
|
|
2343
|
-
<br>
|
|
2344
|
-
<br>
|
|
2345
|
-
<br>
|
|
2346
|
-
<br>
|
|
2347
|
-
<br>
|
|
2348
|
-
<br>
|
|
2349
|
-
|
|
2350
|
-
## Try it
|
|
2351
|
-
|
|
2352
|
-
- <big>`gem install raptor` (Ruby 4.0+)</big>
|
|
2353
|
-
- <big>`bundle exec raptor` to boot your Rack app</big>
|
|
2354
|
-
- <big>CLI mirrors Puma's conventions on purpose</big>
|
|
2355
|
-
- <big>Feedback welcome</big>
|
|
2356
|
-
- <big>Bug reports very welcome</big>
|
|
2357
|
-
|
|
2358
|
-
If you want the long-form of this talk with more diagrams and more depth, the design doc lives at [`docs/raptor-vs-puma.md`](raptor-vs-puma.md) in the repo.
|
|
2359
|
-
|
|
2360
|
-
<br>
|
|
2361
|
-
<br>
|
|
2362
|
-
<br>
|
|
2363
|
-
<br>
|
|
2364
|
-
<br>
|
|
2365
|
-
<br>
|
|
2366
|
-
<br>
|
|
2367
|
-
<br>
|
|
2368
|
-
|
|
2369
|
-
---
|
|
2370
|
-
|
|
2371
|
-
<br>
|
|
2372
|
-
<br>
|
|
2373
|
-
<br>
|
|
2374
|
-
<br>
|
|
2375
|
-
<br>
|
|
2376
|
-
<br>
|
|
2377
|
-
<br>
|
|
2378
|
-
<br>
|
|
2379
|
-
|
|
2380
|
-
## Thanks
|
|
2381
|
-
|
|
2382
|
-
**Raptor**
|
|
2383
|
-
|
|
2384
|
-
[github.com/joshuay03/raptor](https://github.com/joshuay03/raptor)
|
|
2385
|
-
|
|
2386
|
-
**The gems**
|
|
2387
|
-
|
|
2388
|
-
- <big>[atomic-ruby](https://github.com/joshuay03/atomic-ruby)</big>
|
|
2389
|
-
- <big>[ractor-pool](https://github.com/joshuay03/ractor-pool)</big>
|
|
2390
|
-
- <big>[red-black-tree](https://github.com/joshuay03/red-black-tree)</big>
|
|
2391
|
-
- <big>[mmap-ruby](https://github.com/joshuay03/mmap-ruby)</big>
|
|
2392
|
-
- <big>[libbpf-ruby](https://github.com/joshuay03/libbpf-ruby)</big>
|
|
2393
|
-
|
|
2394
|
-
**Design doc**
|
|
2395
|
-
|
|
2396
|
-
[docs/raptor-vs-puma.md](raptor-vs-puma.md)
|
|
2397
|
-
|
|
2398
|
-
_Questions?_
|