radioactive 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- checksums.yaml.gz.sig +0 -0
- data/CHANGELOG.md +85 -0
- data/CLAUDE.md +52 -0
- data/README.md +530 -0
- data/Rakefile +11 -0
- data/Steepfile +24 -0
- data/lib/radioactive/address_check.rb +59 -0
- data/lib/radioactive/errors.rb +28 -0
- data/lib/radioactive/fetcher.rb +355 -0
- data/lib/radioactive/monotonic_clock.rb +9 -0
- data/lib/radioactive/result.rb +5 -0
- data/lib/radioactive/version.rb +5 -0
- data/lib/radioactive.rb +19 -0
- data/lib/tasks/gem.rake +5 -0
- data/lib/tasks/lint/all.rake +11 -0
- data/lib/tasks/lint/rubocop.rake +15 -0
- data/lib/tasks/security.rake +11 -0
- data/lib/tasks/types.rake +16 -0
- data/sig/radioactive.rbs +234 -0
- data/sig/zeitwerk.rbs +13 -0
- data.tar.gz.sig +0 -0
- metadata +112 -0
- metadata.gz.sig +0 -0
data/README.md
ADDED
|
@@ -0,0 +1,530 @@
|
|
|
1
|
+
# Radioactive
|
|
2
|
+
|
|
3
|
+
A hardened HTTP fetcher for Ruby. Safe to point at URLs supplied by untrusted users.
|
|
4
|
+
|
|
5
|
+
## Why
|
|
6
|
+
|
|
7
|
+
If you've ever written code like:
|
|
8
|
+
|
|
9
|
+
```ruby
|
|
10
|
+
URI.open(user_supplied_url).read
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
for a link preview, image proxy, webhook delivery, or metadata extraction, you have a server-side request forgery (SSRF) vulnerability. A malicious user can submit a URL that makes your server fetch:
|
|
14
|
+
|
|
15
|
+
- `http://169.254.169.254/latest/meta-data/iam/security-credentials/` - your AWS instance credentials
|
|
16
|
+
- `http://localhost:6379/` - your Redis, potentially executable
|
|
17
|
+
- `http://10.0.0.5/admin` - your internal admin panel
|
|
18
|
+
- `http://metadata.google.internal/` - your GCP project's tokens
|
|
19
|
+
|
|
20
|
+
`URI.open` happily fetches all of these. So do most general-purpose Ruby HTTP clients (`Net::HTTP`, `Faraday`, `HTTParty`) - your code sees the response body, the attacker gets your secrets.
|
|
21
|
+
|
|
22
|
+
Untrusted URLs also expose you to:
|
|
23
|
+
|
|
24
|
+
- **Slowloris**: a server that drips one byte per second can pin a worker thread.
|
|
25
|
+
- **Response bombs**: a 10 GB response will OOM your process.
|
|
26
|
+
- **Decompression bombs**: 1 KB of gzip can decompress to 100 MB, OOM'ing you anyway.
|
|
27
|
+
- **Redirect-to-internal**: `http://example.com/` → `Location: http://127.0.0.1/` bypasses naive blocklists that only check the original URL.
|
|
28
|
+
- **DNS rebinding**: by the time your address check resolves the hostname, the attacker has flipped the DNS record to `127.0.0.1`.
|
|
29
|
+
|
|
30
|
+
`Radioactive` wraps `Net::HTTP` with defenses for all of these, on by default with zero configuration.
|
|
31
|
+
|
|
32
|
+
## What it protects against
|
|
33
|
+
|
|
34
|
+
| Threat | Default behavior |
|
|
35
|
+
|---------------------------------------------------------------------|------------------------------------------------------------------------------|
|
|
36
|
+
| Cloud-metadata exfiltration (`169.254.169.254`) | Blocked |
|
|
37
|
+
| Loopback (`127.x`, `[::1]`) | Blocked |
|
|
38
|
+
| RFC1918 (`10.x`, `192.168.x`, `172.16-31.x`) | Blocked |
|
|
39
|
+
| IPv6 ULA / link-local / multicast | Blocked |
|
|
40
|
+
| DNS rebinding | Resolved IP is pinned; redirects re-validate the new host |
|
|
41
|
+
| Disallowed schemes (`file://`, `gopher://`, `javascript:`) | Allowlist: `http`, `https` |
|
|
42
|
+
| Embedded credentials (`http://user:pass@host/`) | Rejected |
|
|
43
|
+
| Slowloris / no-read | `read_timeout` per chunk + `total_timeout` |
|
|
44
|
+
| Response bombs | `max_size` enforced per chunk, default 2 MB |
|
|
45
|
+
| Decompression bombs | `Accept-Encoding: identity` default; opt-in gzip bounded on **decoded** size |
|
|
46
|
+
| Redirect chain DoS | `max_redirects`, default 3 |
|
|
47
|
+
| Redirect to private IP | Each hop re-validated through the full pipeline |
|
|
48
|
+
| Header CRLF/NUL injection in caller-supplied headers | Rejected |
|
|
49
|
+
| Non-canonical IP forms (`http://2130706433/`, `http://0x7f000001/`) | Rejected |
|
|
50
|
+
| Hostname-spoof TLS bypass | Pin via `http.ipaddr=` so SNI and cert verification still use the hostname |
|
|
51
|
+
|
|
52
|
+
What it deliberately does **not** do (use Faraday or HTTParty when you control the destination):
|
|
53
|
+
|
|
54
|
+
- POST / PUT / DELETE / PATCH - read-only by design
|
|
55
|
+
- Cookies, sessions, retries, multipart, basic auth
|
|
56
|
+
- HTTP/2, HTTP/3
|
|
57
|
+
- Outbound proxy, connection pooling, HTTP caching, circuit breakers
|
|
58
|
+
|
|
59
|
+
For the full threat model, see [docs/REQUIREMENTS.md](docs/REQUIREMENTS.md).
|
|
60
|
+
|
|
61
|
+
## Installation
|
|
62
|
+
|
|
63
|
+
```bash
|
|
64
|
+
bundle add radioactive
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
Requires Ruby >= 3.2.
|
|
68
|
+
|
|
69
|
+
## Quick start
|
|
70
|
+
|
|
71
|
+
The simplest case mirrors `URI.open`:
|
|
72
|
+
|
|
73
|
+
```ruby
|
|
74
|
+
require "radioactive"
|
|
75
|
+
|
|
76
|
+
body = Radioactive.open("https://example.com/").read
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
For richer access (status, headers, redirect history), use `fetch`:
|
|
80
|
+
|
|
81
|
+
```ruby
|
|
82
|
+
result = Radioactive.fetch("https://example.com/")
|
|
83
|
+
result.status # => 200
|
|
84
|
+
result.body # => "<!doctype html>..."
|
|
85
|
+
result.headers # => {"content-type" => "text/html", ...}
|
|
86
|
+
result.final_url # => #<URI::HTTPS https://example.com/>
|
|
87
|
+
result.hops # => [] (no redirects)
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
If the URL is unsafe (private IP, disallowed scheme, etc.), the fetch raises a `Radioactive::Error`. Most callers in untrusted-input contexts rescue the base class and degrade gracefully:
|
|
91
|
+
|
|
92
|
+
```ruby
|
|
93
|
+
def fetch_metadata(url)
|
|
94
|
+
Radioactive.open(url) do |io|
|
|
95
|
+
Metadata.parse(io.read)
|
|
96
|
+
end
|
|
97
|
+
rescue Radioactive::Error => e
|
|
98
|
+
Rails.logger.warn("metadata fetch refused: #{e.class}: #{e.message}")
|
|
99
|
+
Metadata.empty
|
|
100
|
+
end
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
## Configuration
|
|
104
|
+
|
|
105
|
+
Every option can be passed per-call to `fetch` / `open`, or set on a `Fetcher` instance for reuse.
|
|
106
|
+
|
|
107
|
+
### Smaller body limit (link previews rarely need 2 MB)
|
|
108
|
+
|
|
109
|
+
```ruby
|
|
110
|
+
Radioactive.fetch(url, max_size: 256_000)
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
### Tighter timeouts (interactive request handlers)
|
|
114
|
+
|
|
115
|
+
```ruby
|
|
116
|
+
Radioactive.fetch(url, total_timeout: 5, max_redirects: 1)
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
### Per-tenant / per-context fetcher instances
|
|
120
|
+
|
|
121
|
+
```ruby
|
|
122
|
+
class TenantFetcher
|
|
123
|
+
def initialize(tenant)
|
|
124
|
+
@fetcher = Radioactive::Fetcher.new(
|
|
125
|
+
max_size: tenant.preview_byte_limit,
|
|
126
|
+
max_redirects: 1,
|
|
127
|
+
total_timeout: 8,
|
|
128
|
+
user_agent: "AcmeBot/1.0 (tenant=#{tenant.id})"
|
|
129
|
+
)
|
|
130
|
+
end
|
|
131
|
+
|
|
132
|
+
def call(url) = @fetcher.fetch(url)
|
|
133
|
+
end
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
### Streaming large bodies to disk (low-memory)
|
|
137
|
+
|
|
138
|
+
When you pass a block to `open`, the body streams chunk-by-chunk to a `Tempfile` instead of buffering in memory. Peak per-fetch memory stays at ~16 KB regardless of `max_size`. Three common patterns:
|
|
139
|
+
|
|
140
|
+
**Stream straight to a destination file** - the most common case. The Tempfile yielded to the block is closed and unlinked when the block returns:
|
|
141
|
+
|
|
142
|
+
```ruby
|
|
143
|
+
Radioactive.open(url, max_size: 50_000_000) do |io|
|
|
144
|
+
File.open(destination, "wb") { |dest| IO.copy_stream(io, dest) }
|
|
145
|
+
end
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
**Hash while downloading** - useful for image proxies or content-addressable storage where you want SHA256 of the body without holding the whole thing in memory:
|
|
149
|
+
|
|
150
|
+
```ruby
|
|
151
|
+
require "digest"
|
|
152
|
+
|
|
153
|
+
digest = Digest::SHA256.new
|
|
154
|
+
Radioactive.open(url, max_size: 10_000_000) do |io|
|
|
155
|
+
while (chunk = io.read(64 * 1024))
|
|
156
|
+
digest.update(chunk)
|
|
157
|
+
cache.write_chunk(chunk)
|
|
158
|
+
end
|
|
159
|
+
end
|
|
160
|
+
sha = digest.hexdigest
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
**Compare against `fetch` for the in-memory case** - keep using `fetch` when you actually want the body in a string (link previews, JSON endpoints, anything that fits comfortably under `max_size`):
|
|
164
|
+
|
|
165
|
+
```ruby
|
|
166
|
+
# Small body, want it in memory: use fetch
|
|
167
|
+
result = Radioactive.fetch(url, max_size: 256_000)
|
|
168
|
+
JSON.parse(result.body)
|
|
169
|
+
|
|
170
|
+
# Potentially large body, want to write to disk: use open block form
|
|
171
|
+
Radioactive.open(url, max_size: 50_000_000) do |io|
|
|
172
|
+
File.open(destination, "wb") { |dest| IO.copy_stream(io, dest) }
|
|
173
|
+
end
|
|
174
|
+
```
|
|
175
|
+
|
|
176
|
+
If the body grows past `max_size` during streaming, `Radioactive::SizeError` is raised mid-read and the partially-written Tempfile is unlinked before the exception propagates.
|
|
177
|
+
|
|
178
|
+
## More advanced
|
|
179
|
+
|
|
180
|
+
### Custom address blocklist
|
|
181
|
+
|
|
182
|
+
The default `private_ranges` blocks 25 CIDR ranges (RFC1918, loopback, link-local, IPv6 ULA, etc.). To allow loopback specifically - e.g. when running against a local development server - without disabling all checks:
|
|
183
|
+
|
|
184
|
+
```ruby
|
|
185
|
+
require "ipaddr"
|
|
186
|
+
|
|
187
|
+
ranges = Radioactive::AddressCheck::DEFAULT_PRIVATE_RANGES.reject do |r|
|
|
188
|
+
r.include?(IPAddr.new("127.0.0.1"))
|
|
189
|
+
end
|
|
190
|
+
|
|
191
|
+
Radioactive::Fetcher.new(private_ranges: ranges)
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
For tests, there's an explicit "skip the address check" shortcut. Don't use it in production:
|
|
195
|
+
|
|
196
|
+
```ruby
|
|
197
|
+
Radioactive::Fetcher.new(allow_private: true)
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
### Opting into compressed responses
|
|
201
|
+
|
|
202
|
+
The default `accept_encoding: "identity"` rejects compressed responses to defend against decompression bombs. If you trust the destination enough to opt in (and accept that `max_size` then applies to the *decoded* body):
|
|
203
|
+
|
|
204
|
+
```ruby
|
|
205
|
+
Radioactive.fetch(url, accept_encoding: "gzip")
|
|
206
|
+
```
|
|
207
|
+
|
|
208
|
+
### Custom request headers
|
|
209
|
+
|
|
210
|
+
```ruby
|
|
211
|
+
Radioactive.fetch(url, headers: {
|
|
212
|
+
"Accept" => "application/json",
|
|
213
|
+
"X-Trace-Id" => trace_id
|
|
214
|
+
})
|
|
215
|
+
```
|
|
216
|
+
|
|
217
|
+
`Host`, `User-Agent`, and `Accept-Encoding` are reserved. Other headers go through after CRLF/NUL validation - invalid values raise `SchemeError` before the socket opens.
|
|
218
|
+
|
|
219
|
+
## Errors
|
|
220
|
+
|
|
221
|
+
Every defense raises a distinct subclass of `Radioactive::Error`. Callers in untrusted-input contexts typically rescue the base class:
|
|
222
|
+
|
|
223
|
+
```ruby
|
|
224
|
+
begin
|
|
225
|
+
Radioactive.fetch(user_supplied_url)
|
|
226
|
+
rescue Radioactive::Error => e
|
|
227
|
+
log_and_degrade(e)
|
|
228
|
+
end
|
|
229
|
+
```
|
|
230
|
+
|
|
231
|
+
If you need to handle specific failure modes:
|
|
232
|
+
|
|
233
|
+
| Class | Raised when |
|
|
234
|
+
|---|---|
|
|
235
|
+
| `Radioactive::SchemeError` | Disallowed scheme, missing host, embedded credentials, non-canonical IP literal, or CRLF/NUL in a caller-supplied header |
|
|
236
|
+
| `Radioactive::AddressError` | DNS resolution failed, or any resolved address is in `private_ranges` |
|
|
237
|
+
| `Radioactive::TimeoutError` | `open_timeout`, `read_timeout`, or `total_timeout` exceeded |
|
|
238
|
+
| `Radioactive::SizeError` | `Content-Length` exceeds `max_size`, or the body grows past `max_size` mid-stream |
|
|
239
|
+
| `Radioactive::RedirectError` | `max_redirects` exhausted |
|
|
240
|
+
| `Radioactive::EncodingError` | Server returned an unexpected `Content-Encoding`, or gzip decoding failed |
|
|
241
|
+
| `Radioactive::ResponseError` | Non-2xx status, or transport-level failure (TLS error, connection reset) |
|
|
242
|
+
|
|
243
|
+
`ResponseError` carries `#status`, `#headers`, and `#body` of the partial response when applicable, so you can react to specific HTTP errors:
|
|
244
|
+
|
|
245
|
+
```ruby
|
|
246
|
+
begin
|
|
247
|
+
Radioactive.fetch(url)
|
|
248
|
+
rescue Radioactive::ResponseError => e
|
|
249
|
+
if e.status == 429
|
|
250
|
+
retry_after = e.headers["retry-after"]&.to_i || 60
|
|
251
|
+
sleep retry_after
|
|
252
|
+
retry
|
|
253
|
+
else
|
|
254
|
+
raise
|
|
255
|
+
end
|
|
256
|
+
end
|
|
257
|
+
```
|
|
258
|
+
|
|
259
|
+
A common pattern in URL-shortener / link-preview / image-proxy code paths is to log the failure class and continue with empty data - the user submitted a URL we wouldn't fetch, and that's the end of it:
|
|
260
|
+
|
|
261
|
+
```ruby
|
|
262
|
+
def safe_preview(url)
|
|
263
|
+
result = Radioactive.fetch(url, max_size: 512_000, total_timeout: 5)
|
|
264
|
+
parse_preview(result.body, base: result.final_url)
|
|
265
|
+
rescue Radioactive::AddressError, Radioactive::SchemeError
|
|
266
|
+
# Caller submitted a URL we won't touch (private IP, weird scheme, etc).
|
|
267
|
+
# Don't surface details - just decline.
|
|
268
|
+
nil
|
|
269
|
+
rescue Radioactive::TimeoutError, Radioactive::SizeError, Radioactive::ResponseError => e
|
|
270
|
+
# Caller's URL was reasonable; the destination misbehaved.
|
|
271
|
+
Metrics.increment("preview.refused", tags: {reason: e.class.name.split("::").last})
|
|
272
|
+
nil
|
|
273
|
+
end
|
|
274
|
+
```
|
|
275
|
+
|
|
276
|
+
## Testing
|
|
277
|
+
|
|
278
|
+
When you write tests against code that uses `Radioactive`, you generally want to avoid touching the real network or waiting for real time. The library provides three injection seams for exactly this:
|
|
279
|
+
|
|
280
|
+
| Option | Replaces | Protocol |
|
|
281
|
+
|-----------------------|-----------------------------------------|-----------------------------------------------------------|
|
|
282
|
+
| `resolver:` | `Resolv` (default) | object responding to `getaddresses(host) → Array[String]` |
|
|
283
|
+
| `clock:` | `Radioactive::MonotonicClock` (default) | object responding to `now → Float` |
|
|
284
|
+
| `allow_private: true` | the address blocklist | bool - disables address checks entirely; for tests only |
|
|
285
|
+
|
|
286
|
+
### Stubbing DNS
|
|
287
|
+
|
|
288
|
+
Returning a fixed address per hostname lets you assert exactly what the address-check pipeline sees, without touching real DNS:
|
|
289
|
+
|
|
290
|
+
```ruby
|
|
291
|
+
class StubResolver
|
|
292
|
+
def initialize(map) = @map = map
|
|
293
|
+
|
|
294
|
+
def getaddresses(host)
|
|
295
|
+
@map.fetch(host) { raise "unexpected resolve(#{host.inspect})" }
|
|
296
|
+
end
|
|
297
|
+
end
|
|
298
|
+
|
|
299
|
+
resolver = StubResolver.new(
|
|
300
|
+
"ok.example" => ["8.8.8.8"], # public - passes the address check
|
|
301
|
+
"evil.example" => ["10.0.0.1"] # RFC1918 - blocked by default
|
|
302
|
+
)
|
|
303
|
+
|
|
304
|
+
fetcher = Radioactive::Fetcher.new(resolver: resolver)
|
|
305
|
+
```
|
|
306
|
+
|
|
307
|
+
The "raise on unexpected lookup" pattern is deliberate: it makes tests fail loudly if your code under test resolves a hostname you didn't plan for.
|
|
308
|
+
|
|
309
|
+
### Stubbing the clock
|
|
310
|
+
|
|
311
|
+
A `clock` is anything responding to `now → Float`. To force a `total_timeout` to fire deterministically:
|
|
312
|
+
|
|
313
|
+
```ruby
|
|
314
|
+
class StubClock
|
|
315
|
+
def initialize(values) = @values = values.dup
|
|
316
|
+
def now = @values.shift || raise("clock exhausted")
|
|
317
|
+
end
|
|
318
|
+
|
|
319
|
+
# total_timeout = 5; second clock read happens at t=100, well past the deadline.
|
|
320
|
+
fetcher = Radioactive::Fetcher.new(
|
|
321
|
+
total_timeout: 5,
|
|
322
|
+
clock: StubClock.new([0.0, 100.0])
|
|
323
|
+
)
|
|
324
|
+
|
|
325
|
+
assert_raises(Radioactive::TimeoutError) { fetcher.fetch("https://ok.example/") }
|
|
326
|
+
```
|
|
327
|
+
|
|
328
|
+
### Running against a real local server
|
|
329
|
+
|
|
330
|
+
If your test suite spins up an actual HTTP server (Rack, Sinatra, WEBrick) on `127.0.0.1`, the default `private_ranges` blocks it. Two options:
|
|
331
|
+
|
|
332
|
+
```ruby
|
|
333
|
+
# Coarse: skip address checks entirely
|
|
334
|
+
Radioactive::Fetcher.new(allow_private: true)
|
|
335
|
+
|
|
336
|
+
# Fine: allow loopback only, keep RFC1918 / metadata / etc. blocked
|
|
337
|
+
require "ipaddr"
|
|
338
|
+
ranges = Radioactive::AddressCheck::DEFAULT_PRIVATE_RANGES.reject do |r|
|
|
339
|
+
r.include?(IPAddr.new("127.0.0.1"))
|
|
340
|
+
end
|
|
341
|
+
Radioactive::Fetcher.new(private_ranges: ranges)
|
|
342
|
+
```
|
|
343
|
+
|
|
344
|
+
The fine-grained version is preferable when you want the test to still fail if your code accidentally tries to reach AWS metadata or RFC1918, even from inside the test suite.
|
|
345
|
+
|
|
346
|
+
### Asserting on specific failure modes
|
|
347
|
+
|
|
348
|
+
Each defense raises a distinct `Radioactive::Error` subclass, so tests can pin down exactly *why* a fetch failed. RSpec:
|
|
349
|
+
|
|
350
|
+
```ruby
|
|
351
|
+
expect { Radioactive.fetch("http://10.0.0.1/") }.to raise_error(Radioactive::AddressError)
|
|
352
|
+
expect { Radioactive.fetch("file:///etc/passwd") }.to raise_error(Radioactive::SchemeError)
|
|
353
|
+
expect { Radioactive.fetch("http://2130706433/") }.to raise_error(Radioactive::SchemeError, /numeric host/)
|
|
354
|
+
```
|
|
355
|
+
|
|
356
|
+
Minitest:
|
|
357
|
+
|
|
358
|
+
```ruby
|
|
359
|
+
assert_raises(Radioactive::AddressError) { Radioactive.fetch("http://10.0.0.1/") }
|
|
360
|
+
err = assert_raises(Radioactive::SchemeError) { Radioactive.fetch("http://2130706433/") }
|
|
361
|
+
assert_match(/numeric host/, err.message)
|
|
362
|
+
```
|
|
363
|
+
|
|
364
|
+
### Testing streaming downloads
|
|
365
|
+
|
|
366
|
+
If your code uses the `open` block form to stream a body to disk, two things are worth asserting in tests: the right bytes ended up where you expected, *and* that the size cap actually trips when the body is too large. A pattern with a stub resolver and a `Rack`-style local server (using `allow_private: true` for the test):
|
|
367
|
+
|
|
368
|
+
```ruby
|
|
369
|
+
def test_image_proxy_streams_to_disk
|
|
370
|
+
destination = Tempfile.new(["proxied", ".bin"])
|
|
371
|
+
|
|
372
|
+
Radioactive.open(@server_url, allow_private: true, max_size: 10_000_000) do |io|
|
|
373
|
+
IO.copy_stream(io, destination)
|
|
374
|
+
end
|
|
375
|
+
|
|
376
|
+
destination.rewind
|
|
377
|
+
assert_equal expected_bytesize, destination.size
|
|
378
|
+
assert_equal expected_sha, Digest::SHA256.file(destination.path).hexdigest
|
|
379
|
+
ensure
|
|
380
|
+
destination&.close
|
|
381
|
+
destination&.unlink
|
|
382
|
+
end
|
|
383
|
+
|
|
384
|
+
def test_image_proxy_refuses_oversize_body
|
|
385
|
+
# Server returns 500 KB; we cap at 100 KB.
|
|
386
|
+
assert_raises(Radioactive::SizeError) do
|
|
387
|
+
Radioactive.open(@server_url, allow_private: true, max_size: 100_000) do |io|
|
|
388
|
+
io.read
|
|
389
|
+
end
|
|
390
|
+
end
|
|
391
|
+
end
|
|
392
|
+
```
|
|
393
|
+
|
|
394
|
+
The block-form `open` unlinks its Tempfile on every exit path - success or `SizeError` - so tests don't have to manage cleanup of *its* internal Tempfile, only their own.
|
|
395
|
+
|
|
396
|
+
### Testing Radioactive itself
|
|
397
|
+
|
|
398
|
+
If you're contributing, the test layout is:
|
|
399
|
+
|
|
400
|
+
- `test/test_*.rb` - Minitest tests, auto-discovered. Each file groups tests by component (`test_address_check.rb`, `test_fetcher.rb` for unit-level, `test_fetcher_http.rb` for integration).
|
|
401
|
+
- `test/support/server_fixture.rb` - small `TCPServer`-based HTTP fixture used by the integration tests. Two modes:
|
|
402
|
+
- `TestServer.run(handler)` - handler is `(method, path, headers) → response`, where response is either a raw HTTP response string or a `{status:, headers:, body:, chunked:}` hash.
|
|
403
|
+
- `TestServer.run { |client, method, path, headers| ... }` - block writes directly to the client socket, for tests that need wire-level control (e.g. dripping bytes for slowloris).
|
|
404
|
+
|
|
405
|
+
Two layers of tests:
|
|
406
|
+
|
|
407
|
+
- **Unit** (no socket): scheme/credential validation, IP blocklist, dual-A rejection, `total_timeout` deadline trip - all use stubbed `resolver` and/or `clock`.
|
|
408
|
+
- **Integration** (real socket): redirect handling, body size cap, `Content-Length` pre-reject, `Content-Encoding` rejection, gzip decode + decoded-size cap, `read_timeout`, slowloris, redirect-to-private revalidation.
|
|
409
|
+
|
|
410
|
+
Useful commands:
|
|
411
|
+
|
|
412
|
+
```bash
|
|
413
|
+
bundle exec rake # full default: test + lint + types
|
|
414
|
+
bundle exec rake test # tests only
|
|
415
|
+
bundle exec rake test TESTOPTS="--name=/redirect/" # filter by test name
|
|
416
|
+
bundle exec ruby -Ilib -Itest test/test_fetcher.rb # run one test file directly
|
|
417
|
+
```
|
|
418
|
+
|
|
419
|
+
## Development
|
|
420
|
+
|
|
421
|
+
After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
|
|
422
|
+
|
|
423
|
+
To install this gem onto your local machine, run `bundle exec rake gem:install`. See [Releasing](#releasing) below for publishing a new version.
|
|
424
|
+
|
|
425
|
+
## Releasing
|
|
426
|
+
|
|
427
|
+
A release cuts a new version, tags it in git, and pushes the `.gem` to [rubygems.org](https://rubygems.org). The published gem requires MFA for any push (set in `radioactive.gemspec`), so you'll be prompted for a one-time code.
|
|
428
|
+
|
|
429
|
+
### Pre-flight
|
|
430
|
+
|
|
431
|
+
1. `bundle exec rake` exits 0. Tests, lint, and type checks all green.
|
|
432
|
+
2. `lib/radioactive/version.rb` bumped to the new version following [SemVer](https://semver.org).
|
|
433
|
+
3. `CHANGELOG.md` has a populated section for the new version with a date, and `[Unreleased]` is empty (or its contents have been moved into the new section).
|
|
434
|
+
4. Working tree is clean (`rake gem:release` enforces this).
|
|
435
|
+
5. You have an active rubygems session: `gem signin` if `gem whoami` returns nothing.
|
|
436
|
+
|
|
437
|
+
### Cutting the release
|
|
438
|
+
|
|
439
|
+
```bash
|
|
440
|
+
bundle exec rake gem:release
|
|
441
|
+
```
|
|
442
|
+
|
|
443
|
+
This runs, in order:
|
|
444
|
+
|
|
445
|
+
1. `gem:release:guard_clean` - aborts if there are uncommitted changes.
|
|
446
|
+
2. `gem:build` - builds `pkg/radioactive-X.Y.Z.gem`.
|
|
447
|
+
3. `gem:release:source_control_push` - creates `git tag vX.Y.Z` and pushes the commit + tag.
|
|
448
|
+
4. `gem:release:rubygem_push` - pushes the `.gem` to rubygems.org (this is the step that prompts for your MFA code).
|
|
449
|
+
|
|
450
|
+
If anything fails midway (e.g. push rejected, MFA timeout), the tag may already be pushed but the gem may not be published. Re-run `bundle exec rake gem:release:rubygem_push` to retry just the publish step.
|
|
451
|
+
|
|
452
|
+
### After the release
|
|
453
|
+
|
|
454
|
+
- Open `CHANGELOG.md` and start a fresh empty `## [Unreleased]` section above the just-released version.
|
|
455
|
+
- Bump `lib/radioactive/version.rb` to the next anticipated version with a `.dev` or `.alpha` suffix if you want subsequent local builds to be distinguishable from the release.
|
|
456
|
+
- Push that commit; future changes accumulate under `[Unreleased]` until the next release.
|
|
457
|
+
|
|
458
|
+
### Stronger publishing setup (optional)
|
|
459
|
+
|
|
460
|
+
For a security-focused gem, consider [RubyGems Trusted Publishing](https://guides.rubygems.org/trusted-publishing/) instead of pushing from a developer machine: a tagged commit triggers a GitHub Actions workflow that authenticates to rubygems via OIDC and publishes without any long-lived API key on disk. Removes the "stolen laptop = compromised gem" risk and complements the MFA requirement.
|
|
461
|
+
|
|
462
|
+
## Type checking (RBS + Steep)
|
|
463
|
+
|
|
464
|
+
This gem ships type signatures in `sig/radioactive.rbs` and uses [Steep](https://github.com/soutaro/steep) to verify them. If you're new to Ruby type checking, here's what each piece is doing.
|
|
465
|
+
|
|
466
|
+
### The 30-second mental model
|
|
467
|
+
|
|
468
|
+
- **RBS** is a separate file format that says *what* your code looks like (method names, parameter types, return types). It's just declarations - like a header file - and it does **not** affect how Ruby runs. The actual sigs live in `sig/radioactive.rbs`.
|
|
469
|
+
- **Steep** is a type checker. It reads `lib/*.rb` and compares it to `sig/*.rbs`, and complains when the two disagree (e.g. a method's actual return type doesn't match what the sig promised).
|
|
470
|
+
|
|
471
|
+
So: `sig/` is a contract; Steep verifies the contract. Neither runs in production - both are dev-time tools.
|
|
472
|
+
|
|
473
|
+
### Why we bother
|
|
474
|
+
|
|
475
|
+
1. **API drift protection.** If we rename a public method or change a return type and forget to update the sig, `rake types` fails. The sig file becomes load-bearing instead of decorative.
|
|
476
|
+
2. **Better editor support for users.** Consumers of the gem who use Steep or RBS-aware editors (RubyMine, VS Code with Sorbet/Steep extensions) get autocomplete and inline errors when they call `Radioactive.fetch(...)` with the wrong arguments.
|
|
477
|
+
|
|
478
|
+
### The commands you'll use
|
|
479
|
+
|
|
480
|
+
| Command | What it does |
|
|
481
|
+
|-----------------------------------|--------------------------------------------------------------------------------------------------------------|
|
|
482
|
+
| `bundle exec rake types:validate` | Sanity-checks the RBS file itself - catches typos like referring to a class that doesn't exist. Fast. |
|
|
483
|
+
| `bundle exec rake types:check` | Runs Steep, which compares `lib/` against `sig/`. **This is the one that catches real drift.** Slower (~5s). |
|
|
484
|
+
| `bundle exec rake types` | Runs both. Also part of the default `rake` task, so `bundle exec rake` now runs `test` + `lint` + `types`. |
|
|
485
|
+
|
|
486
|
+
### When do I need to touch `sig/radioactive.rbs`?
|
|
487
|
+
|
|
488
|
+
Whenever the **public API** changes:
|
|
489
|
+
|
|
490
|
+
- Adding/removing/renaming a public method (`Radioactive.foo`, `Fetcher#bar`, etc.)
|
|
491
|
+
- Adding/removing/renaming a configuration option (the kwargs on `Fetcher.new`, `fetch`, `open`)
|
|
492
|
+
- Changing what a public method returns or accepts
|
|
493
|
+
|
|
494
|
+
You usually do **not** need to update sigs for private-method changes - the `private` block in the sig is loosely typed and just exists so Steep stops complaining about internal call sites. If you add a brand-new private method and Steep grumbles "Method X is not declared," add a line for it (with `untyped` for unknown types if you're not sure).
|
|
495
|
+
|
|
496
|
+
### When Steep complains
|
|
497
|
+
|
|
498
|
+
The most common failure modes:
|
|
499
|
+
|
|
500
|
+
- **"Cannot pass a value of type X as an argument of type Y"** - your code passes the wrong type somewhere. Either fix the code or, if the code is correct and the sig is too narrow, widen the sig.
|
|
501
|
+
- **"Method X is not declared in RBS"** - you added a method but didn't add a sig line. Add one (in the `private` block if it's internal).
|
|
502
|
+
- **"Cannot find the declaration of constant: `Foo`"** - Steep doesn't know about a class from a stdlib or external gem. If it's stdlib, add `library "foo"` to `Steepfile`. If it's a gem, you'll need an inline stub (see `sig/zeitwerk.rbs` for an example).
|
|
503
|
+
|
|
504
|
+
When in doubt, paste the error into a search engine - Steep's diagnostic IDs (e.g. `Ruby::ArgumentTypeMismatch`) make for good search terms.
|
|
505
|
+
|
|
506
|
+
### Files involved
|
|
507
|
+
|
|
508
|
+
- `sig/radioactive.rbs` - the signatures themselves.
|
|
509
|
+
- `sig/zeitwerk.rbs` - a tiny stub for the one external gem we use at the entry point.
|
|
510
|
+
- `Steepfile` - Steep's config (which signatures to load, which Ruby files to check).
|
|
511
|
+
- `lib/tasks/types.rake` - the rake tasks.
|
|
512
|
+
|
|
513
|
+
## Contributing
|
|
514
|
+
|
|
515
|
+
Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/radioactive.
|
|
516
|
+
|
|
517
|
+
### Before opening a PR
|
|
518
|
+
|
|
519
|
+
1. **Run the full check locally.** `bundle exec rake` runs tests, RuboCop, and the type check (`rbs validate` + `steep`). All three must pass.
|
|
520
|
+
2. **Update the spec if you're changing behavior.** The contract lives at [docs/REQUIREMENTS.md](docs/REQUIREMENTS.md). Code and spec drift is a bug - fix them in the same commit.
|
|
521
|
+
3. **Update the signatures if you're changing the public API.** Any change to a public method on `Radioactive`, `Fetcher`, `Result`, or any error class needs a matching update in `sig/radioactive.rbs`. The Type checking section above covers what to do; `bundle exec rake types` will tell you when something's off.
|
|
522
|
+
4. **Add a test.** Especially for security-sensitive paths: every defense in the threat model should have a test that proves the failure path actually closes. See `test/test_fetcher.rb` and `test/test_fetcher_http.rb` for patterns.
|
|
523
|
+
|
|
524
|
+
### Security issues
|
|
525
|
+
|
|
526
|
+
Please report security-sensitive issues privately rather than via a public GitHub issue. (TODO: contact email or GitHub Security Advisory link.)
|
|
527
|
+
|
|
528
|
+
### Style
|
|
529
|
+
|
|
530
|
+
The project uses [`standard`](https://github.com/standardrb/standard) (via RuboCop) for formatting. `bundle exec rake lint` reports issues; `bundle exec rake lint:rubocop:autocorrect` fixes the autocorrectable ones.
|
data/Rakefile
ADDED
data/Steepfile
ADDED
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
D = Steep::Diagnostic
|
|
4
|
+
|
|
5
|
+
target :lib do
|
|
6
|
+
signature "sig"
|
|
7
|
+
check "lib"
|
|
8
|
+
|
|
9
|
+
# Stdlib signatures we use directly. Steep ships these with the rbs gem.
|
|
10
|
+
library "uri"
|
|
11
|
+
library "ipaddr"
|
|
12
|
+
library "stringio"
|
|
13
|
+
library "tempfile"
|
|
14
|
+
library "net-http"
|
|
15
|
+
library "openssl"
|
|
16
|
+
library "resolv"
|
|
17
|
+
library "zlib"
|
|
18
|
+
|
|
19
|
+
# Empty hash/array literals would force every internal variable initialization
|
|
20
|
+
# to carry an inline RBS annotation comment. Silence — they're advisory.
|
|
21
|
+
configure_code_diagnostics do |hash|
|
|
22
|
+
hash[D::Ruby::UnannotatedEmptyCollection] = nil
|
|
23
|
+
end
|
|
24
|
+
end
|
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "ipaddr"
|
|
4
|
+
|
|
5
|
+
module Radioactive
|
|
6
|
+
module AddressCheck
|
|
7
|
+
DEFAULT_PRIVATE_RANGES = %w[
|
|
8
|
+
0.0.0.0/8
|
|
9
|
+
10.0.0.0/8
|
|
10
|
+
100.64.0.0/10
|
|
11
|
+
127.0.0.0/8
|
|
12
|
+
169.254.0.0/16
|
|
13
|
+
172.16.0.0/12
|
|
14
|
+
192.0.0.0/24
|
|
15
|
+
192.0.2.0/24
|
|
16
|
+
192.168.0.0/16
|
|
17
|
+
198.18.0.0/15
|
|
18
|
+
198.51.100.0/24
|
|
19
|
+
203.0.113.0/24
|
|
20
|
+
224.0.0.0/4
|
|
21
|
+
240.0.0.0/4
|
|
22
|
+
255.255.255.255/32
|
|
23
|
+
::/128
|
|
24
|
+
::1/128
|
|
25
|
+
64:ff9b::/96
|
|
26
|
+
100::/64
|
|
27
|
+
2001::/32
|
|
28
|
+
2001:db8::/32
|
|
29
|
+
fc00::/7
|
|
30
|
+
fe80::/10
|
|
31
|
+
ff00::/8
|
|
32
|
+
].map { |cidr| IPAddr.new(cidr) }.freeze
|
|
33
|
+
|
|
34
|
+
module_function
|
|
35
|
+
|
|
36
|
+
# @param ip [IPAddr]
|
|
37
|
+
# @param ranges [Array<IPAddr>]
|
|
38
|
+
def forbidden?(ip, ranges = DEFAULT_PRIVATE_RANGES)
|
|
39
|
+
candidate = ip.ipv4_mapped? ? ip.native : ip
|
|
40
|
+
ranges.any? { |r| r.include?(candidate) }
|
|
41
|
+
end
|
|
42
|
+
|
|
43
|
+
# @param host [String]
|
|
44
|
+
# @param resolver [#getaddresses]
|
|
45
|
+
# @return [Array<IPAddr>]
|
|
46
|
+
def resolve(host, resolver)
|
|
47
|
+
begin
|
|
48
|
+
ip = IPAddr.new(host)
|
|
49
|
+
return [ip]
|
|
50
|
+
rescue IPAddr::Error
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
addresses = Array(resolver.getaddresses(host))
|
|
54
|
+
addresses.map { |a| IPAddr.new(a) }
|
|
55
|
+
rescue IPAddr::Error => e
|
|
56
|
+
raise AddressError, "could not parse resolved address: #{e.message}"
|
|
57
|
+
end
|
|
58
|
+
end
|
|
59
|
+
end
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Radioactive
|
|
4
|
+
class Error < StandardError; end
|
|
5
|
+
|
|
6
|
+
class SchemeError < Error; end
|
|
7
|
+
|
|
8
|
+
class AddressError < Error; end
|
|
9
|
+
|
|
10
|
+
class TimeoutError < Error; end
|
|
11
|
+
|
|
12
|
+
class SizeError < Error; end
|
|
13
|
+
|
|
14
|
+
class RedirectError < Error; end
|
|
15
|
+
|
|
16
|
+
class EncodingError < Error; end
|
|
17
|
+
|
|
18
|
+
class ResponseError < Error
|
|
19
|
+
attr_reader :status, :headers, :body
|
|
20
|
+
|
|
21
|
+
def initialize(message = nil, status: nil, headers: nil, body: nil)
|
|
22
|
+
super(message)
|
|
23
|
+
@status = status
|
|
24
|
+
@headers = headers
|
|
25
|
+
@body = body
|
|
26
|
+
end
|
|
27
|
+
end
|
|
28
|
+
end
|