otto 2.6.0 → 2.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.github/workflows/ci.yml +1 -1
- data/.github/workflows/claude-code-review.yml +1 -1
- data/.github/workflows/claude.yml +1 -1
- data/.github/workflows/code-smells.yml +2 -2
- data/.github/workflows/release-gem.yml +1 -1
- data/.github/workflows/ruby-lint.yml +1 -1
- data/.github/workflows/yardoc.yml +1 -1
- data/.pre-commit-config.yaml +22 -5
- data/CHANGELOG.rst +254 -0
- data/Gemfile +2 -1
- data/Gemfile.lock +13 -11
- data/README.md +13 -3
- data/docs/.gitignore +1 -0
- data/docs/1108-STREAMING_ARCHITECTURE_ANALYSIS.md +1105 -0
- data/docs/1108-STREAMING_SUPPORT_SUMMARY.md +376 -0
- data/docs/geo-country.md +180 -0
- data/docs/migrating/v2.3.0.md +55 -22
- data/docs/reverse-proxy-network-services.md +19 -6
- data/examples/simple_geo_resolver.rb +38 -5
- data/lib/otto/caddy_tls/localhost_guard.rb +43 -25
- data/lib/otto/core/middleware_stack.rb +72 -25
- data/lib/otto/env_keys.rb +58 -12
- data/lib/otto/logging_helpers.rb +50 -1
- data/lib/otto/mcp/rate_limiting.rb +5 -2
- data/lib/otto/privacy/config.rb +245 -3
- data/lib/otto/privacy/core.rb +104 -14
- data/lib/otto/privacy/geo_resolver.rb +228 -128
- data/lib/otto/privacy/ip_privacy.rb +24 -0
- data/lib/otto/privacy/redacted_fingerprint.rb +54 -2
- data/lib/otto/privacy.rb +3 -1
- data/lib/otto/request.rb +25 -9
- data/lib/otto/security/authentication/auth_failure.rb +36 -2
- data/lib/otto/security/authentication/auth_strategy.rb +12 -2
- data/lib/otto/security/authentication/authorization_failure.rb +7 -0
- data/lib/otto/security/authentication/route_auth_wrapper.rb +138 -31
- data/lib/otto/security/config.rb +61 -1
- data/lib/otto/security/core.rb +4 -1
- data/lib/otto/security/csp/report_middleware.rb +3 -1
- data/lib/otto/security/middleware/ip_privacy_middleware.rb +228 -18
- data/lib/otto/security/rate_limiter.rb +7 -1
- data/lib/otto/utils.rb +100 -0
- data/lib/otto/version.rb +1 -1
- data/lib/otto.rb +11 -3
- metadata +5 -2
|
@@ -0,0 +1,376 @@
|
|
|
1
|
+
# Otto Streaming Support: Executive Summary
|
|
2
|
+
|
|
3
|
+
**Investigation Date**: 2025-11-08
|
|
4
|
+
**Question**: Should Otto support Server-Sent Events (SSE) and WebSockets?
|
|
5
|
+
**Answer**: **No** - Use separate services or long-polling instead
|
|
6
|
+
|
|
7
|
+
> **Note**: The Ruby snippets below illustrate the recommended architecture. They
|
|
8
|
+
> are written against the current Otto Logic-class contract
|
|
9
|
+
> (`initialize(context, params, locale)` + `process`), but have not been executed
|
|
10
|
+
> end-to-end. Runnable examples are tracked separately.
|
|
11
|
+
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
## Quick Recommendation
|
|
15
|
+
|
|
16
|
+
| Use Case | Solution | Complexity | Otto Compatible? |
|
|
17
|
+
|----------|----------|------------|------------------|
|
|
18
|
+
| **Low-frequency updates (<1/min)** | Long-polling | ⭐ Simple | ✅ Yes |
|
|
19
|
+
| **Medium-frequency updates (1-10/sec)** | Separate SSE service | ⭐⭐ Moderate | ✅ Via integration |
|
|
20
|
+
| **High-frequency updates (>10/sec)** | Separate WebSocket service | ⭐⭐⭐ Complex | ✅ Via integration |
|
|
21
|
+
| **Bidirectional communication** | Separate WebSocket service | ⭐⭐⭐ Complex | ✅ Via integration |
|
|
22
|
+
|
|
23
|
+
---
|
|
24
|
+
|
|
25
|
+
## Key Findings
|
|
26
|
+
|
|
27
|
+
### 1. **Otto's Architecture is Fundamentally Incompatible with Streaming**
|
|
28
|
+
|
|
29
|
+
Otto is designed as a **stateless, synchronous, request/response** framework:
|
|
30
|
+
|
|
31
|
+
```
|
|
32
|
+
Request → Middleware → Route → Handler → Response → Close Connection
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
SSE/WebSocket require **stateful, long-lived connections**:
|
|
36
|
+
|
|
37
|
+
```
|
|
38
|
+
Request → Upgrade → Keep Open → Stream Data (minutes/hours) → Close
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
**Incompatibilities**:
|
|
42
|
+
- ❌ Response handlers expect complete responses (not streaming enumerators)
|
|
43
|
+
- ❌ Middleware stack can't unwind during long-lived connections
|
|
44
|
+
- ❌ Configuration freezing prevents runtime streaming adjustments
|
|
45
|
+
- ❌ Requires async servers (Falcon, Iodine) - breaks server-agnostic design
|
|
46
|
+
- ❌ Stateful routing complicates horizontal scaling
|
|
47
|
+
|
|
48
|
+
### 2. **Industry Best Practice: Separate Services**
|
|
49
|
+
|
|
50
|
+
Modern frameworks separate real-time communication from REST APIs:
|
|
51
|
+
|
|
52
|
+
**Rails (ActionCable)**:
|
|
53
|
+
```
|
|
54
|
+
Rails App (Puma) → Redis Pub/Sub ← ActionCable (Separate Process)
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
**Node.js (Express + Socket.IO)**:
|
|
58
|
+
```
|
|
59
|
+
Express (HTTP Routes) + Socket.IO (Separate Layer)
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
**Benefits**:
|
|
63
|
+
- ✅ Independent scaling (scale WebSocket separately from API)
|
|
64
|
+
- ✅ Technology choice (use best tool for each job)
|
|
65
|
+
- ✅ Fault isolation (WebSocket crash doesn't affect API)
|
|
66
|
+
- ✅ Clear architectural boundaries
|
|
67
|
+
|
|
68
|
+
### 3. **Long-Polling Works Perfectly with Otto**
|
|
69
|
+
|
|
70
|
+
For low-to-medium frequency updates, long-polling is **simple and effective**:
|
|
71
|
+
|
|
72
|
+
```ruby
|
|
73
|
+
# Otto route (works with any Rack server)
|
|
74
|
+
GET /api/notifications/poll NotificationPollLogic response=json auth=session
|
|
75
|
+
|
|
76
|
+
class NotificationPollLogic
|
|
77
|
+
attr_reader :context, :params, :locale
|
|
78
|
+
|
|
79
|
+
def initialize(context, params, locale)
|
|
80
|
+
@context = context
|
|
81
|
+
@params = params
|
|
82
|
+
@locale = locale
|
|
83
|
+
end
|
|
84
|
+
|
|
85
|
+
def process
|
|
86
|
+
# Logic-class params are string-keyed (LogicClassHandler does not apply
|
|
87
|
+
# Otto::Static.indifferent_params).
|
|
88
|
+
timeout = params['timeout'].to_i.clamp(1, 30)
|
|
89
|
+
last_id = params['last_id'].to_i
|
|
90
|
+
start_time = Time.now
|
|
91
|
+
|
|
92
|
+
# Identity comes from the StrategyResult, never from request params.
|
|
93
|
+
user_id = context.user_id
|
|
94
|
+
|
|
95
|
+
loop do
|
|
96
|
+
notifications = fetch_new(user_id, last_id)
|
|
97
|
+
return { notifications: notifications } if notifications.any?
|
|
98
|
+
|
|
99
|
+
break if Time.now - start_time > timeout
|
|
100
|
+
|
|
101
|
+
sleep 0.5
|
|
102
|
+
end
|
|
103
|
+
|
|
104
|
+
{ notifications: [] }
|
|
105
|
+
end
|
|
106
|
+
end
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
**Benefits**:
|
|
110
|
+
- ✅ HTTP-based (cacheable, proxy-friendly, standard tooling)
|
|
111
|
+
- ✅ Works with Otto's synchronous model
|
|
112
|
+
- ✅ Compatible with any Rack server (Puma, Unicorn, Passenger)
|
|
113
|
+
- ✅ Simple debugging (standard HTTP requests/responses)
|
|
114
|
+
- ✅ No external dependencies
|
|
115
|
+
|
|
116
|
+
---
|
|
117
|
+
|
|
118
|
+
## Recommended Solutions
|
|
119
|
+
|
|
120
|
+
### Option 1: Long-Polling (SIMPLEST)
|
|
121
|
+
|
|
122
|
+
**When to use**:
|
|
123
|
+
- Updates less than 1 per minute
|
|
124
|
+
- Moderate concurrency (<10,000 clients)
|
|
125
|
+
- Simple deployment preferred
|
|
126
|
+
|
|
127
|
+
**Example**: see the `NotificationPollLogic` sketch above.
|
|
128
|
+
|
|
129
|
+
**Complexity**: ⭐ Simple
|
|
130
|
+
**Otto Integration**: ✅ Native support (no changes needed)
|
|
131
|
+
|
|
132
|
+
---
|
|
133
|
+
|
|
134
|
+
### Option 2: Separate SSE Service (RECOMMENDED FOR REAL-TIME)
|
|
135
|
+
|
|
136
|
+
**When to use**:
|
|
137
|
+
- Updates 1-10 per second
|
|
138
|
+
- High concurrency (>10,000 clients)
|
|
139
|
+
- Near-instant updates required (<100ms latency)
|
|
140
|
+
|
|
141
|
+
**Architecture**:
|
|
142
|
+
```
|
|
143
|
+
┌─────────────────┐
|
|
144
|
+
│ Otto API │ ← Stateless HTTP (authentication, business logic)
|
|
145
|
+
│ (Puma) │
|
|
146
|
+
└─────────────────┘
|
|
147
|
+
↓
|
|
148
|
+
┌─────────┐
|
|
149
|
+
│ Redis │ ← Message queue (pub/sub)
|
|
150
|
+
│ Pub/Sub │
|
|
151
|
+
└─────────┘
|
|
152
|
+
↓
|
|
153
|
+
┌─────────────────┐
|
|
154
|
+
│ SSE Service │ ← Stateful streaming (Falcon/Iodine)
|
|
155
|
+
│ (Falcon) │
|
|
156
|
+
└─────────────────┘
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
**Example**: see "Otto + Falcon SSE Integration" in
|
|
160
|
+
`docs/1108-STREAMING_ARCHITECTURE_ANALYSIS.md`.
|
|
161
|
+
|
|
162
|
+
**Complexity**: ⭐⭐ Moderate
|
|
163
|
+
**Otto Integration**: ✅ Via Redis pub/sub
|
|
164
|
+
|
|
165
|
+
---
|
|
166
|
+
|
|
167
|
+
### Option 3: Third-Party Service (COMMERCIAL)
|
|
168
|
+
|
|
169
|
+
**When to use**:
|
|
170
|
+
- Don't want to manage WebSocket infrastructure
|
|
171
|
+
- Need global CDN distribution
|
|
172
|
+
- Require guaranteed SLA
|
|
173
|
+
|
|
174
|
+
**Options**:
|
|
175
|
+
- **Mercure**: Open-source SSE hub (self-hosted or managed)
|
|
176
|
+
- **Ably**: Commercial real-time messaging platform
|
|
177
|
+
- **Pusher**: Commercial WebSocket/SSE service
|
|
178
|
+
|
|
179
|
+
**Complexity**: ⭐⭐ Moderate (integration)
|
|
180
|
+
**Otto Integration**: ✅ Via HTTP API
|
|
181
|
+
|
|
182
|
+
---
|
|
183
|
+
|
|
184
|
+
## Documentation Created
|
|
185
|
+
|
|
186
|
+
### **docs/1108-STREAMING_ARCHITECTURE_ANALYSIS.md** (15,000+ words)
|
|
187
|
+
Comprehensive technical analysis covering:
|
|
188
|
+
- Otto's current architecture (detailed lifecycle analysis)
|
|
189
|
+
- Technical requirements for SSE/WebSocket (Rack 3, hijacking, etc.)
|
|
190
|
+
- Industry patterns (Rails, Sinatra, Roda, Go, Node.js)
|
|
191
|
+
- Compatibility analysis (why it doesn't fit)
|
|
192
|
+
- Best practices and anti-patterns
|
|
193
|
+
- Detailed recommendations with code examples
|
|
194
|
+
|
|
195
|
+
### Runnable examples: not yet published
|
|
196
|
+
|
|
197
|
+
Draft `examples/otto_falcon_sse_integration.rb` and
|
|
198
|
+
`examples/long_polling_example.rb` were written alongside this analysis but
|
|
199
|
+
targeted an Otto API surface that does not exist (`Otto::RequestContext`,
|
|
200
|
+
`enable_sessions!`, `raise_concern`, a `session` helper, `Rack::Handler::Puma`
|
|
201
|
+
under the Rack 3 pin) and used symbol-keyed params that Logic routes never
|
|
202
|
+
populate. They were withheld rather than shipped as copy-paste material; the
|
|
203
|
+
architecture they demonstrated is preserved in the snippets here and in the
|
|
204
|
+
analysis document.
|
|
205
|
+
|
|
206
|
+
---
|
|
207
|
+
|
|
208
|
+
## Key Insights
|
|
209
|
+
|
|
210
|
+
### The Real Question
|
|
211
|
+
|
|
212
|
+
**Not**: "Can Otto support SSE/WebSocket?"
|
|
213
|
+
(Technically possible with massive refactoring)
|
|
214
|
+
|
|
215
|
+
**But**: "Should Otto support SSE/WebSocket?"
|
|
216
|
+
(Architecturally inadvisable)
|
|
217
|
+
|
|
218
|
+
### Answer: **No**
|
|
219
|
+
|
|
220
|
+
Otto should remain focused on its core strengths:
|
|
221
|
+
- ✅ **Stateless** HTTP APIs
|
|
222
|
+
- ✅ **Security-first** design (CSRF, rate limiting, validation)
|
|
223
|
+
- ✅ **Privacy by default** (IP masking, geo-location)
|
|
224
|
+
- ✅ **Server-agnostic** (works with any Rack server)
|
|
225
|
+
- ✅ **Simple** and predictable architecture
|
|
226
|
+
|
|
227
|
+
Adding SSE/WebSocket would:
|
|
228
|
+
- ❌ Compromise architectural integrity
|
|
229
|
+
- ❌ Force specific async servers (Falcon, Iodine)
|
|
230
|
+
- ❌ Complicate security guarantees (middleware assumptions broken)
|
|
231
|
+
- ❌ Add significant complexity for niche use case
|
|
232
|
+
- ❌ Go against industry best practices (separation of concerns)
|
|
233
|
+
|
|
234
|
+
---
|
|
235
|
+
|
|
236
|
+
## What Otto SHOULD Do
|
|
237
|
+
|
|
238
|
+
### 1. ✅ Document Integration Patterns
|
|
239
|
+
|
|
240
|
+
Add official guide: "Integrating Otto with Real-Time Services"
|
|
241
|
+
- Long-polling patterns (built-in support)
|
|
242
|
+
- Separate SSE service pattern (Otto + Falcon + Redis)
|
|
243
|
+
- Third-party service integration (Mercure, Ably, Pusher)
|
|
244
|
+
|
|
245
|
+
### 2. ✅ Provide Example Code
|
|
246
|
+
|
|
247
|
+
Add to `examples/` directory, written against the real Logic-class contract and
|
|
248
|
+
verified to boot:
|
|
249
|
+
- `long_polling_example.rb`
|
|
250
|
+
- `otto_falcon_sse_integration.rb`
|
|
251
|
+
- `otto_mercure_integration.rb`
|
|
252
|
+
|
|
253
|
+
### 3. ⚠️ Consider Plugin System (If Community Demands)
|
|
254
|
+
|
|
255
|
+
**Only if there's strong demand**, create experimental plugin:
|
|
256
|
+
- Clearly marked "experimental" and "unsupported"
|
|
257
|
+
- Requires Falcon/Iodine (documented)
|
|
258
|
+
- Security implications documented
|
|
259
|
+
- No core changes required
|
|
260
|
+
|
|
261
|
+
### 4. ❌ Do NOT Add to Core
|
|
262
|
+
|
|
263
|
+
Preserve Otto's architectural integrity by:
|
|
264
|
+
- Keeping core stateless and synchronous
|
|
265
|
+
- Maintaining server-agnostic design
|
|
266
|
+
- Focusing on security and simplicity
|
|
267
|
+
- Following industry best practices (separation of concerns)
|
|
268
|
+
|
|
269
|
+
---
|
|
270
|
+
|
|
271
|
+
## Migration Guide for Existing Users
|
|
272
|
+
|
|
273
|
+
If you currently need real-time updates:
|
|
274
|
+
|
|
275
|
+
### Step 1: Assess Your Use Case
|
|
276
|
+
|
|
277
|
+
**Low-frequency updates (<1/min)**:
|
|
278
|
+
→ Use long-polling (Otto native support)
|
|
279
|
+
|
|
280
|
+
**Medium-frequency updates (1-10/sec)**:
|
|
281
|
+
→ Use separate SSE service (Otto + Falcon + Redis)
|
|
282
|
+
|
|
283
|
+
**High-frequency or bidirectional**:
|
|
284
|
+
→ Use separate WebSocket service or commercial solution
|
|
285
|
+
|
|
286
|
+
### Step 2: Implementation Path
|
|
287
|
+
|
|
288
|
+
#### For Long-Polling:
|
|
289
|
+
1. Create Otto route with long-polling logic
|
|
290
|
+
2. Use `sleep` loop with timeout
|
|
291
|
+
3. Client polls with timeout parameter
|
|
292
|
+
4. No external dependencies needed
|
|
293
|
+
|
|
294
|
+
See the `NotificationPollLogic` sketch above.
|
|
295
|
+
|
|
296
|
+
#### For Separate SSE Service:
|
|
297
|
+
1. Otto API handles authentication and publishes to Redis
|
|
298
|
+
2. Separate Falcon app subscribes to Redis and streams SSE
|
|
299
|
+
3. Client connects to SSE service with JWT token from Otto
|
|
300
|
+
4. Scale Otto and SSE services independently
|
|
301
|
+
|
|
302
|
+
See "Otto + Falcon SSE Integration" in
|
|
303
|
+
`docs/1108-STREAMING_ARCHITECTURE_ANALYSIS.md`.
|
|
304
|
+
|
|
305
|
+
### Step 3: Deployment
|
|
306
|
+
|
|
307
|
+
**Long-Polling**:
|
|
308
|
+
- Deploy with existing Otto setup (Puma, Unicorn, Passenger)
|
|
309
|
+
- Increase thread pool size for long-polling routes
|
|
310
|
+
- Monitor connection pool (ensure enough threads)
|
|
311
|
+
|
|
312
|
+
**Separate SSE Service**:
|
|
313
|
+
- Deploy Otto API with Puma (standard)
|
|
314
|
+
- Deploy Falcon SSE service separately (dedicated servers)
|
|
315
|
+
- Use Redis for pub/sub (cluster-ready)
|
|
316
|
+
- Configure Nginx with sticky sessions for SSE
|
|
317
|
+
- Scale services independently based on load
|
|
318
|
+
|
|
319
|
+
---
|
|
320
|
+
|
|
321
|
+
## Performance Guidance
|
|
322
|
+
|
|
323
|
+
### Long-Polling Capacity
|
|
324
|
+
|
|
325
|
+
**Example**: Puma with 5 workers × 32 threads = 160 concurrent requests
|
|
326
|
+
|
|
327
|
+
If long-polling uses 30s timeout:
|
|
328
|
+
- 160 concurrent connections
|
|
329
|
+
- ~320 clients with 50% utilization
|
|
330
|
+
- Up to 10,000 clients with proper thread tuning
|
|
331
|
+
|
|
332
|
+
**Good for**: Dashboard metrics, low-volume notifications
|
|
333
|
+
|
|
334
|
+
### SSE/WebSocket Capacity
|
|
335
|
+
|
|
336
|
+
**Example**: Falcon with 4 workers (async)
|
|
337
|
+
|
|
338
|
+
Each worker handles thousands of concurrent connections via fibers:
|
|
339
|
+
- 10,000+ concurrent SSE connections per server
|
|
340
|
+
- Horizontal scaling via Redis pub/sub
|
|
341
|
+
- Near-instant message delivery
|
|
342
|
+
|
|
343
|
+
**Good for**: Chat, multiplayer, high-frequency updates
|
|
344
|
+
|
|
345
|
+
---
|
|
346
|
+
|
|
347
|
+
## Conclusion
|
|
348
|
+
|
|
349
|
+
**Otto should NOT integrate SSE/WebSocket support** because:
|
|
350
|
+
|
|
351
|
+
1. **Architectural mismatch**: Stateless vs stateful paradigms
|
|
352
|
+
2. **Industry consensus**: Separate services is best practice
|
|
353
|
+
3. **Complexity cost**: Massive refactoring for niche use case
|
|
354
|
+
4. **Better alternatives**: Long-polling (simple) or separate services (powerful)
|
|
355
|
+
|
|
356
|
+
**Instead, Otto should**:
|
|
357
|
+
|
|
358
|
+
1. ✅ Document long-polling patterns (works today)
|
|
359
|
+
2. ✅ Provide integration examples (Otto + Falcon + Redis)
|
|
360
|
+
3. ✅ Recommend third-party solutions (Mercure, Ably, Pusher)
|
|
361
|
+
4. ✅ Stay focused on stateless HTTP APIs
|
|
362
|
+
|
|
363
|
+
**This preserves Otto's core strengths** while enabling users who need real-time functionality to integrate appropriate solutions.
|
|
364
|
+
|
|
365
|
+
---
|
|
366
|
+
|
|
367
|
+
## Further Reading
|
|
368
|
+
|
|
369
|
+
- **Technical Analysis**: `docs/1108-STREAMING_ARCHITECTURE_ANALYSIS.md`
|
|
370
|
+
- **Rack 3 Streaming**: https://github.com/rack/rack/issues/1600
|
|
371
|
+
- **ActionCable Architecture**: https://guides.rubyonrails.org/action_cable_overview.html
|
|
372
|
+
- **SSE vs WebSocket**: https://ably.com/blog/websockets-vs-sse
|
|
373
|
+
|
|
374
|
+
---
|
|
375
|
+
|
|
376
|
+
**End of Summary**
|
data/docs/geo-country.md
ADDED
|
@@ -0,0 +1,180 @@
|
|
|
1
|
+
# Geo-country resolution
|
|
2
|
+
|
|
3
|
+
Otto resolves a country-level ISO 3166-1 alpha-2 code for each request and
|
|
4
|
+
exposes it as `req.geo_country` / `env['otto.privacy.geo_country']`. Resolution
|
|
5
|
+
is country-only by design — that is the privacy posture; there is no city or
|
|
6
|
+
region lookup.
|
|
7
|
+
|
|
8
|
+
## Resolution order
|
|
9
|
+
|
|
10
|
+
`Otto::Privacy::GeoResolver.resolve` returns the first hit from:
|
|
11
|
+
|
|
12
|
+
1. **Application-configured header** (`geo_header:`) — e.g. `X-Client-Country`.
|
|
13
|
+
2. **Known provider headers** — Cloudflare (`CF-IPCountry`), AWS CloudFront,
|
|
14
|
+
Fastly, Akamai Edgescape, Azure Front Door, **Vercel**
|
|
15
|
+
(`X-Vercel-IP-Country`), and a few semi-standard names
|
|
16
|
+
(`X-Geo-Country`, `X-Country-Code`, `Country-Code`).
|
|
17
|
+
3. **Custom resolver** (`GeoResolver.custom_resolver`) — your own callable.
|
|
18
|
+
Unlike the other geo settings, this is **class-level** (see
|
|
19
|
+
[Configuration](#configuration)).
|
|
20
|
+
4. **Local MMDB database** (`geo_db_path:` / `geo_db_reader:`) — a MaxMind-DB
|
|
21
|
+
country database.
|
|
22
|
+
5. **`'**'`** — the unknown sentinel, when nothing else matches.
|
|
23
|
+
|
|
24
|
+
Steps 1 and 2 are **only consulted when geo headers can be trusted** (see
|
|
25
|
+
[Header trust](#header-trust-and-spoofing) below).
|
|
26
|
+
|
|
27
|
+
Resolution is **honest**: Otto does not guess from a hardcoded IP-range table.
|
|
28
|
+
When no header, custom resolver, or database resolves a country, the result is
|
|
29
|
+
`'**'`.
|
|
30
|
+
|
|
31
|
+
### Privacy: masked IP and masked env
|
|
32
|
+
|
|
33
|
+
The database lookup in step 4 runs on the request's **masked** IP
|
|
34
|
+
(e.g. `203.0.113.0`), never the real address. `check_geo_database` masks the IP
|
|
35
|
+
internally with the config's `octet_precision` before the lookup, so even a
|
|
36
|
+
direct `GeoResolver.resolve` caller passing a real IP does not expose it to the
|
|
37
|
+
database. Country-level MMDB networks are almost always ≥ /24, so the default
|
|
38
|
+
/24-masked value (`octet_precision: 1`) resolves to the same country.
|
|
39
|
+
|
|
40
|
+
In the middleware path Otto additionally hands `resolve` a **masked env view**:
|
|
41
|
+
`REMOTE_ADDR`, `X-Forwarded-For`, `X-Real-IP`, `X-Client-IP`, and the RFC 7239
|
|
42
|
+
`Forwarded` header are masked. So a `custom_resolver` cannot read the raw client
|
|
43
|
+
IP out of `env` either — use the `ip` argument (already masked), not `env`.
|
|
44
|
+
|
|
45
|
+
> **`octet_precision: 2`** masks two octets (a /16). That is coarser than most
|
|
46
|
+
> country networks, so it can reduce database hit rate for the small share of
|
|
47
|
+
> countries whose ranges are finer than /16 — those requests fall through to
|
|
48
|
+
> `'**'`. Header and custom-resolver sources are unaffected (they ignore the
|
|
49
|
+
> IP). Keep the default precision if you rely on the MMDB fallback.
|
|
50
|
+
|
|
51
|
+
## Configuration
|
|
52
|
+
|
|
53
|
+
All geo configuration is **boot-time only** (set once during single-threaded
|
|
54
|
+
initialization, before serving requests), matching `custom_resolver`'s
|
|
55
|
+
contract. `geo_header`, `geo_db_path`, and `geo_db_reader` are stored on the
|
|
56
|
+
instance's `Otto::Privacy::Config`, so separate Otto instances hold independent
|
|
57
|
+
geo configuration.
|
|
58
|
+
|
|
59
|
+
> **`custom_resolver` is the exception — it is class-level, not per-instance.**
|
|
60
|
+
> `GeoResolver.custom_resolver=` sets a singleton on the `GeoResolver` class, so
|
|
61
|
+
> it is **shared across every Otto instance in the process** (last write wins).
|
|
62
|
+
> If you run multiple Otto instances that need different resolver strategies,
|
|
63
|
+
> the custom resolver cannot distinguish them — branch inside a single resolver
|
|
64
|
+
> on `env`, or use per-instance `geo_db_reader` instead.
|
|
65
|
+
|
|
66
|
+
```ruby
|
|
67
|
+
otto.configure_ip_privacy(
|
|
68
|
+
geo: true, # default; false disables geo entirely
|
|
69
|
+
geo_header: 'X-Client-Country', # trusted app header (optional)
|
|
70
|
+
geo_db_path: 'data/geo-whois-asn-country.mmdb', # local MMDB fallback (optional)
|
|
71
|
+
# geo_db_reader: MaxMind::DB.new(path), # or bring your own reader (optional)
|
|
72
|
+
)
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
- **`geo: false`** short-circuits everything: no header reads, and any loaded
|
|
76
|
+
database is unloaded from memory (`req.geo_country` becomes `nil`).
|
|
77
|
+
- **`geo_header:`** accepts either the HTTP header name (`X-Client-Country`) or
|
|
78
|
+
the Rack CGI env key (`HTTP_X_CLIENT_COUNTRY`), in any case, and is
|
|
79
|
+
canonicalized to the env-key form. Pass `''` to clear.
|
|
80
|
+
- **`geo_db_path:`** is loaded once at boot in `MODE_MEMORY`. An unreadable
|
|
81
|
+
path, a corrupt/non-MMDB file, or a missing `maxmind-db` gem raises
|
|
82
|
+
`ArgumentError` **at configuration time**, not per-request. Pass `''` to
|
|
83
|
+
unload.
|
|
84
|
+
- **`geo_db_reader:`** injects any object responding to `#get(ip)` (a
|
|
85
|
+
preconfigured `MaxMind::DB` reader or a test double), keeping the reader and
|
|
86
|
+
data-source choice independent of Otto. It **overrides** `geo_db_path` when
|
|
87
|
+
both are given in the same call; supplying `geo_db_path` alone in a later call
|
|
88
|
+
clears a prior reader override.
|
|
89
|
+
|
|
90
|
+
Each keyword follows a `nil` = "leave unchanged" contract; pass `''` to a header
|
|
91
|
+
or path to clear it. Any geo-affecting change triggers the boot-time database
|
|
92
|
+
(re)load, so a bad `geo_db_path` fails at the `configure_ip_privacy` call.
|
|
93
|
+
|
|
94
|
+
## The database: gem and datafile
|
|
95
|
+
|
|
96
|
+
The reader and the data file are independent — the MMDB format is the interop
|
|
97
|
+
point.
|
|
98
|
+
|
|
99
|
+
### Reader gem (`maxmind-db`)
|
|
100
|
+
|
|
101
|
+
The [`maxmind-db`](https://rubygems.org/gems/maxmind-db) gem (official MaxMind
|
|
102
|
+
reader, Apache-2.0, pure Ruby, zero runtime deps) is an **optional**
|
|
103
|
+
dependency. Otto only `require`s it when a database is configured. Add it to
|
|
104
|
+
your app when you use the database fallback:
|
|
105
|
+
|
|
106
|
+
```ruby
|
|
107
|
+
# Gemfile
|
|
108
|
+
gem 'maxmind-db', '~> 1.4'
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
### Data file (`geo-whois-asn-country`)
|
|
112
|
+
|
|
113
|
+
The recommended data file is
|
|
114
|
+
[`geo-whois-asn-country`](https://github.com/sapics/ip-location-db) from
|
|
115
|
+
sapics/ip-location-db: **PDDL v1.0 (public domain, no attribution required)**,
|
|
116
|
+
rebuilt daily, shipped as MMDB. Otto vendors no database — country data goes
|
|
117
|
+
stale, and a public-domain file you refresh on your own schedule keeps
|
|
118
|
+
licensing and freshness in your control.
|
|
119
|
+
|
|
120
|
+
Download it (IPv4+IPv6) into a path of your choosing:
|
|
121
|
+
|
|
122
|
+
```bash
|
|
123
|
+
mkdir -p data
|
|
124
|
+
curl -fsSL -o data/geo-whois-asn-country.mmdb \
|
|
125
|
+
https://github.com/sapics/ip-location-db/releases/download/latest/geo-whois-asn-country.mmdb
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
Refresh it on your own schedule (e.g. a daily cron job running the same curl).
|
|
129
|
+
Any MMDB country database works — GeoLite2-Country, DB-IP Country Lite,
|
|
130
|
+
iplocate, etc. — since `GeoResolver` tolerates the record shapes country
|
|
131
|
+
databases actually use: nested `country.iso_code` (GeoLite2-Country style), a
|
|
132
|
+
flat `country_code` string, and a bare-string `country`.
|
|
133
|
+
|
|
134
|
+
> **Note on GeoLite2:** its EULA requires a MaxMind account/license key and
|
|
135
|
+
> obliges consumers to refresh within 30 days of each release. A PDDL dataset
|
|
136
|
+
> avoids both obligations.
|
|
137
|
+
|
|
138
|
+
## Header trust and spoofing
|
|
139
|
+
|
|
140
|
+
Every geo header is trivially client-spoofable unless the request actually
|
|
141
|
+
arrived through the CDN that sets it. Otto trusts geo headers — both the
|
|
142
|
+
configured `geo_header` and the provider headers — **only** for a request that
|
|
143
|
+
demonstrably arrived via a configured **CIDR trusted proxy**
|
|
144
|
+
(`env['otto.via_trusted_proxy']` with `trusted_proxies` configured). A spoofed
|
|
145
|
+
header on a direct connection is ignored, and resolution falls through to the
|
|
146
|
+
custom resolver / database.
|
|
147
|
+
|
|
148
|
+
Origins Otto cannot verify are **not** trusted:
|
|
149
|
+
|
|
150
|
+
- **No trusted-proxy configuration.** A direct internet client could otherwise
|
|
151
|
+
pick its own country by sending `CF-IPCountry` / `X-Client-Country`, so with
|
|
152
|
+
no `trusted_proxies` configured, header steps are skipped and resolution falls
|
|
153
|
+
to the resolver / database (`'**'` if neither is set).
|
|
154
|
+
- **Count-based `trusted_proxy_depth` mode.** The header-setting hop cannot be
|
|
155
|
+
verified as a geo-CDN, so depth mode does not enable header trust. This
|
|
156
|
+
conflict fails loud: configuring a `geo_header` together with a
|
|
157
|
+
`trusted_proxy_depth` raises `ArgumentError` at configuration time (in
|
|
158
|
+
either order) instead of silently ignoring the header per-request.
|
|
159
|
+
Database-backed geo remains fully supported under depth. The built-in
|
|
160
|
+
provider headers stay legal (there is nothing to configure, so nothing
|
|
161
|
+
can raise) but are equally inert — header trust requires CIDR-verified
|
|
162
|
+
proxies — so only an explicitly configured `geo_header` is rejected, and
|
|
163
|
+
depth deployments that want geo should set `geo_db_path`.
|
|
164
|
+
|
|
165
|
+
**Migration:** to keep header-based geo, configure `trusted_proxies` (CIDR
|
|
166
|
+
matchers) so Otto can verify the proxy origin. Depth-mode and header-only
|
|
167
|
+
deployments should set `geo_db_path` for a local database instead; otherwise
|
|
168
|
+
resolution returns `'**'`.
|
|
169
|
+
|
|
170
|
+
## Acceptance behavior summary
|
|
171
|
+
|
|
172
|
+
| Scenario | Result |
|
|
173
|
+
| --- | --- |
|
|
174
|
+
| Configured `geo_header` present and trusted | wins over provider headers |
|
|
175
|
+
| Request not via a verified CIDR trusted proxy | geo headers skipped |
|
|
176
|
+
| No `trusted_proxies` configured | geo headers skipped (not trusted) |
|
|
177
|
+
| Database lookup | uses the masked IP only |
|
|
178
|
+
| `geo: false` | `nil`, no database in memory |
|
|
179
|
+
| Bad `geo_db_path` | raises at boot, not per-request |
|
|
180
|
+
| Nothing matches | `'**'` |
|
data/docs/migrating/v2.3.0.md
CHANGED
|
@@ -55,8 +55,16 @@ return `false` behind a TLS-terminating trusted proxy.
|
|
|
55
55
|
|
|
56
56
|
The middleware now records the peer-trust decision once (before masking) in a
|
|
57
57
|
leak-free boolean `env['otto.via_trusted_proxy']`, and `secure?` reads it.
|
|
58
|
-
|
|
59
|
-
|
|
58
|
+
Since the first release after 2.7.0 the key is **tri-state**: it is written
|
|
59
|
+
only when proxy trust is actually configured (CIDR matchers or a depth), so a
|
|
60
|
+
present key is authoritative in both directions and an *absent* key means "no
|
|
61
|
+
proxy trust configured". When the key is absent (standalone request use, or an
|
|
62
|
+
unconfigured deployment), `secure?` falls back to the same decision the
|
|
63
|
+
middleware would have implied: a CIDR check of the connecting peer — or, since
|
|
64
|
+
the depth-mode peer-trust fix
|
|
65
|
+
([#226](https://github.com/delano/otto/issues/226), first release after
|
|
66
|
+
2.7.0), an unconditional grant when `trusted_proxy_depth` is configured. No
|
|
67
|
+
app changes are required.
|
|
60
68
|
|
|
61
69
|
### 3. Privacy helpers now return values (previously `nil`)
|
|
62
70
|
|
|
@@ -100,7 +108,7 @@ use `Otto::Request` standalone without the Otto middleware stack.
|
|
|
100
108
|
| Key | Type | Meaning |
|
|
101
109
|
|-----|------|---------|
|
|
102
110
|
| `otto.client_ip` | String | Canonical client IP, resolved once. Masked when privacy is enabled; resolved real IP when disabled or exempt. Read by `Request#ip` / `#client_ipaddress`. |
|
|
103
|
-
| `otto.via_trusted_proxy` | Boolean |
|
|
111
|
+
| `otto.via_trusted_proxy` | Boolean (tri-state: may be absent) | Peer trust decided before masking, written **only when proxy trust is configured** (first release after 2.7.0): `true` on a CIDR match — or unconditionally when depth mode is configured ([#226](https://github.com/delano/otto/issues/226)); `false` means trust is configured and the peer failed it (authoritative deny). Absent = no proxy trust configured — the only case for consumer-side fallback heuristics. Read by `Request#secure?`. |
|
|
104
112
|
|
|
105
113
|
The privacy data keys remain `otto.privacy.{fingerprint,masked_ip,hashed_ip,geo_country}`.
|
|
106
114
|
|
|
@@ -187,15 +195,26 @@ them). Which *multi-hop* header depth counts from — `X-Forwarded-For` (default
|
|
|
187
195
|
the RFC 7239 `Forwarded` header, or `Both` — is configurable as of 2.3.1; see
|
|
188
196
|
*Selecting the forwarded header* below.
|
|
189
197
|
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
|
|
198
|
+
**Depth grants peer trust ([#226](https://github.com/delano/otto/issues/226),
|
|
199
|
+
first release after 2.7.0).** Configuring a depth asserts that the connecting
|
|
200
|
+
peer *is* your proxy tier — that is what the setting means — so depth mode
|
|
201
|
+
records `env['otto.via_trusted_proxy'] = true` on every request, and
|
|
202
|
+
`Otto::Request#secure?` (which consults that flag) honors a forwarded
|
|
203
|
+
`X-Forwarded-Proto` / `X-Scheme` in depth mode. Releases up to and including
|
|
204
|
+
2.7.0 behaved differently: the flag was derived solely from the
|
|
205
|
+
`trusted_proxies` CIDR identity check, which is always empty under depth
|
|
206
|
+
(the modes are mutually exclusive), so it was recorded `false` and `secure?`
|
|
207
|
+
reflected only a direct TLS connection (`HTTPS=on` / port 443). Because the
|
|
208
|
+
grant is unconditional, the *origin lockdown* prerequisite below now covers
|
|
209
|
+
proto trust exactly as it covers IP resolution. Geo headers are unaffected:
|
|
210
|
+
they remain gated on enumerated `trusted_proxies` matchers and are still
|
|
211
|
+
**not** trusted in depth mode (a hop trusted by count cannot be verified as a
|
|
212
|
+
geo-setting CDN). As of the first release after 2.7.0 this carve-out fails
|
|
213
|
+
loud instead of silently: configuring an ip-privacy `geo_header` together
|
|
214
|
+
with a `trusted_proxy_depth` raises `ArgumentError` at configuration time
|
|
215
|
+
(in either order, with a freeze-time backstop) — use filter mode for
|
|
216
|
+
header-based geo, or a geo database (`geo_db_path`) under depth. Database-
|
|
217
|
+
backed geo with depth remains fully supported.
|
|
199
218
|
|
|
200
219
|
### Selecting the forwarded header (added in 2.3.1)
|
|
201
220
|
|
|
@@ -249,9 +268,10 @@ Otto.new(routes, trusted_proxy_depth: 1, trusted_proxy_header: 'Forwarded')
|
|
|
249
268
|
This is the inherent trade-off versus CIDR-walk: depth relies on a fixed network
|
|
250
269
|
**topology** instead of enumerable proxy **addresses**. If a client can reach
|
|
251
270
|
your app directly (origin not locked down), it can pad `X-Forwarded-For` so that
|
|
252
|
-
a forged value lands at `chain[-(N+1)]`, spoofing the resolved client IP. (
|
|
253
|
-
|
|
254
|
-
|
|
271
|
+
a forged value lands at `chain[-(N+1)]`, spoofing the resolved client IP. (Since
|
|
272
|
+
[#226](https://github.com/delano/otto/issues/226) this applies to proto trust
|
|
273
|
+
too: depth grants `otto.via_trusted_proxy`, so a directly-reachable origin
|
|
274
|
+
could spoof the scheme via `X-Forwarded-Proto` as well as the client IP.)
|
|
255
275
|
|
|
256
276
|
Before enabling depth, ensure the origin only accepts connections from the proxy
|
|
257
277
|
tier (private networking, security groups, an authenticating header the proxy
|
|
@@ -262,13 +282,26 @@ injects, etc.). If you can enumerate your proxies instead, prefer CIDR-walk.
|
|
|
262
282
|
If you are collapsing OneTimeSecret's `ClientIpHelpers` / `ConfigureTrustedProxy`
|
|
263
283
|
depth logic onto this resolver, note two intentional differences:
|
|
264
284
|
|
|
265
|
-
- **
|
|
266
|
-
**plus** `REMOTE_ADDR`,
|
|
267
|
-
|
|
268
|
-
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
|
|
285
|
+
- **Map depth values directly — do NOT add one.** Otto's chain is
|
|
286
|
+
`X-Forwarded-For` **plus** `REMOTE_ADDR`, and the client is selected at
|
|
287
|
+
`chain[-(N+1)]` — the appended peer is already accounted for by the index
|
|
288
|
+
arithmetic. `trusted_proxy_depth = N` therefore means "N proxy hops,
|
|
289
|
+
counting the connecting peer as hop 1", which is exactly what the
|
|
290
|
+
operator-facing OTS `depth: N` documents ("1 = standard single reverse
|
|
291
|
+
proxy"). Map **`trusted_proxy_depth = ots_depth`**, and keep a parity
|
|
292
|
+
regression test on the OTS side — including a padded-chain case asserting a
|
|
293
|
+
forged leftmost `X-Forwarded-For` entry is never selected.
|
|
294
|
+
|
|
295
|
+
> **Correction.** Earlier revisions of this guide recommended
|
|
296
|
+
> `trusted_proxy_depth = ots_depth + 1` to "keep existing `depth:` values'
|
|
297
|
+
> meaning". That reproduced an internal off-by-one of the deleted OTS
|
|
298
|
+
> walker (whose indexed path selected `XFF[-(depth+1)]`, one position left
|
|
299
|
+
> of its own documented contract), not the operator-facing meaning. Under
|
|
300
|
+
> the `+1` remap every honest documented-topology request hit the
|
|
301
|
+
> short-chain fallback and resolved the **proxy** address as the client,
|
|
302
|
+
> and a single forged leftmost `X-Forwarded-For` entry re-lengthened the
|
|
303
|
+
> chain so the forged value was selected. The direct mapping resolves the
|
|
304
|
+
> true client on honest chains and is padding-resistant.
|
|
272
305
|
|
|
273
306
|
- **Stricter short-chain behavior (kept on purpose).** When the chain is shorter
|
|
274
307
|
than `N + 1`, Otto returns `REMOTE_ADDR` (the peer), whereas OTS returned the
|