otto 2.6.0 → 2.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.github/workflows/ci.yml +1 -1
- data/.github/workflows/claude-code-review.yml +1 -1
- data/.github/workflows/claude.yml +1 -1
- data/.github/workflows/code-smells.yml +2 -2
- data/.github/workflows/release-gem.yml +1 -1
- data/.github/workflows/ruby-lint.yml +1 -1
- data/.github/workflows/yardoc.yml +1 -1
- data/.pre-commit-config.yaml +22 -5
- data/CHANGELOG.rst +218 -0
- data/Gemfile +2 -1
- data/Gemfile.lock +12 -10
- data/README.md +13 -3
- data/docs/.gitignore +1 -0
- data/docs/1108-STREAMING_ARCHITECTURE_ANALYSIS.md +1105 -0
- data/docs/1108-STREAMING_SUPPORT_SUMMARY.md +376 -0
- data/docs/geo-country.md +172 -0
- data/docs/reverse-proxy-network-services.md +19 -6
- data/examples/simple_geo_resolver.rb +38 -5
- data/lib/otto/caddy_tls/localhost_guard.rb +43 -25
- data/lib/otto/core/middleware_stack.rb +72 -25
- data/lib/otto/env_keys.rb +32 -0
- data/lib/otto/logging_helpers.rb +50 -1
- data/lib/otto/mcp/rate_limiting.rb +5 -2
- data/lib/otto/privacy/config.rb +245 -3
- data/lib/otto/privacy/core.rb +93 -14
- data/lib/otto/privacy/geo_resolver.rb +228 -128
- data/lib/otto/privacy/ip_privacy.rb +24 -0
- data/lib/otto/privacy/redacted_fingerprint.rb +54 -2
- data/lib/otto/privacy.rb +3 -1
- data/lib/otto/request.rb +8 -1
- data/lib/otto/security/authentication/auth_failure.rb +36 -2
- data/lib/otto/security/authentication/auth_strategy.rb +12 -2
- data/lib/otto/security/authentication/authorization_failure.rb +7 -0
- data/lib/otto/security/authentication/route_auth_wrapper.rb +138 -31
- data/lib/otto/security/config.rb +23 -1
- data/lib/otto/security/core.rb +4 -1
- data/lib/otto/security/csp/report_middleware.rb +3 -1
- data/lib/otto/security/middleware/ip_privacy_middleware.rb +201 -12
- data/lib/otto/security/rate_limiter.rb +7 -1
- data/lib/otto/utils.rb +100 -0
- data/lib/otto/version.rb +1 -1
- data/lib/otto.rb +11 -3
- metadata +6 -6
|
@@ -0,0 +1,376 @@
|
|
|
1
|
+
# Otto Streaming Support: Executive Summary
|
|
2
|
+
|
|
3
|
+
**Investigation Date**: 2025-11-08
|
|
4
|
+
**Question**: Should Otto support Server-Sent Events (SSE) and WebSockets?
|
|
5
|
+
**Answer**: **No** - Use separate services or long-polling instead
|
|
6
|
+
|
|
7
|
+
> **Note**: The Ruby snippets below illustrate the recommended architecture. They
|
|
8
|
+
> are written against the current Otto Logic-class contract
|
|
9
|
+
> (`initialize(context, params, locale)` + `process`), but have not been executed
|
|
10
|
+
> end-to-end. Runnable examples are tracked separately.
|
|
11
|
+
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
## Quick Recommendation
|
|
15
|
+
|
|
16
|
+
| Use Case | Solution | Complexity | Otto Compatible? |
|
|
17
|
+
|----------|----------|------------|------------------|
|
|
18
|
+
| **Low-frequency updates (<1/min)** | Long-polling | ⭐ Simple | ✅ Yes |
|
|
19
|
+
| **Medium-frequency updates (1-10/sec)** | Separate SSE service | ⭐⭐ Moderate | ✅ Via integration |
|
|
20
|
+
| **High-frequency updates (>10/sec)** | Separate WebSocket service | ⭐⭐⭐ Complex | ✅ Via integration |
|
|
21
|
+
| **Bidirectional communication** | Separate WebSocket service | ⭐⭐⭐ Complex | ✅ Via integration |
|
|
22
|
+
|
|
23
|
+
---
|
|
24
|
+
|
|
25
|
+
## Key Findings
|
|
26
|
+
|
|
27
|
+
### 1. **Otto's Architecture is Fundamentally Incompatible with Streaming**
|
|
28
|
+
|
|
29
|
+
Otto is designed as a **stateless, synchronous, request/response** framework:
|
|
30
|
+
|
|
31
|
+
```
|
|
32
|
+
Request → Middleware → Route → Handler → Response → Close Connection
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
SSE/WebSocket require **stateful, long-lived connections**:
|
|
36
|
+
|
|
37
|
+
```
|
|
38
|
+
Request → Upgrade → Keep Open → Stream Data (minutes/hours) → Close
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
**Incompatibilities**:
|
|
42
|
+
- ❌ Response handlers expect complete responses (not streaming enumerators)
|
|
43
|
+
- ❌ Middleware stack can't unwind during long-lived connections
|
|
44
|
+
- ❌ Configuration freezing prevents runtime streaming adjustments
|
|
45
|
+
- ❌ Requires async servers (Falcon, Iodine) - breaks server-agnostic design
|
|
46
|
+
- ❌ Stateful routing complicates horizontal scaling
|
|
47
|
+
|
|
48
|
+
### 2. **Industry Best Practice: Separate Services**
|
|
49
|
+
|
|
50
|
+
Modern frameworks separate real-time communication from REST APIs:
|
|
51
|
+
|
|
52
|
+
**Rails (ActionCable)**:
|
|
53
|
+
```
|
|
54
|
+
Rails App (Puma) → Redis Pub/Sub ← ActionCable (Separate Process)
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
**Node.js (Express + Socket.IO)**:
|
|
58
|
+
```
|
|
59
|
+
Express (HTTP Routes) + Socket.IO (Separate Layer)
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
**Benefits**:
|
|
63
|
+
- ✅ Independent scaling (scale WebSocket separately from API)
|
|
64
|
+
- ✅ Technology choice (use best tool for each job)
|
|
65
|
+
- ✅ Fault isolation (WebSocket crash doesn't affect API)
|
|
66
|
+
- ✅ Clear architectural boundaries
|
|
67
|
+
|
|
68
|
+
### 3. **Long-Polling Works Perfectly with Otto**
|
|
69
|
+
|
|
70
|
+
For low-to-medium frequency updates, long-polling is **simple and effective**:
|
|
71
|
+
|
|
72
|
+
```ruby
|
|
73
|
+
# Otto route (works with any Rack server)
|
|
74
|
+
GET /api/notifications/poll NotificationPollLogic response=json auth=session
|
|
75
|
+
|
|
76
|
+
class NotificationPollLogic
|
|
77
|
+
attr_reader :context, :params, :locale
|
|
78
|
+
|
|
79
|
+
def initialize(context, params, locale)
|
|
80
|
+
@context = context
|
|
81
|
+
@params = params
|
|
82
|
+
@locale = locale
|
|
83
|
+
end
|
|
84
|
+
|
|
85
|
+
def process
|
|
86
|
+
# Logic-class params are string-keyed (LogicClassHandler does not apply
|
|
87
|
+
# Otto::Static.indifferent_params).
|
|
88
|
+
timeout = params['timeout'].to_i.clamp(1, 30)
|
|
89
|
+
last_id = params['last_id'].to_i
|
|
90
|
+
start_time = Time.now
|
|
91
|
+
|
|
92
|
+
# Identity comes from the StrategyResult, never from request params.
|
|
93
|
+
user_id = context.user_id
|
|
94
|
+
|
|
95
|
+
loop do
|
|
96
|
+
notifications = fetch_new(user_id, last_id)
|
|
97
|
+
return { notifications: notifications } if notifications.any?
|
|
98
|
+
|
|
99
|
+
break if Time.now - start_time > timeout
|
|
100
|
+
|
|
101
|
+
sleep 0.5
|
|
102
|
+
end
|
|
103
|
+
|
|
104
|
+
{ notifications: [] }
|
|
105
|
+
end
|
|
106
|
+
end
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
**Benefits**:
|
|
110
|
+
- ✅ HTTP-based (cacheable, proxy-friendly, standard tooling)
|
|
111
|
+
- ✅ Works with Otto's synchronous model
|
|
112
|
+
- ✅ Compatible with any Rack server (Puma, Unicorn, Passenger)
|
|
113
|
+
- ✅ Simple debugging (standard HTTP requests/responses)
|
|
114
|
+
- ✅ No external dependencies
|
|
115
|
+
|
|
116
|
+
---
|
|
117
|
+
|
|
118
|
+
## Recommended Solutions
|
|
119
|
+
|
|
120
|
+
### Option 1: Long-Polling (SIMPLEST)
|
|
121
|
+
|
|
122
|
+
**When to use**:
|
|
123
|
+
- Updates less than 1 per minute
|
|
124
|
+
- Moderate concurrency (<10,000 clients)
|
|
125
|
+
- Simple deployment preferred
|
|
126
|
+
|
|
127
|
+
**Example**: see the `NotificationPollLogic` sketch above.
|
|
128
|
+
|
|
129
|
+
**Complexity**: ⭐ Simple
|
|
130
|
+
**Otto Integration**: ✅ Native support (no changes needed)
|
|
131
|
+
|
|
132
|
+
---
|
|
133
|
+
|
|
134
|
+
### Option 2: Separate SSE Service (RECOMMENDED FOR REAL-TIME)
|
|
135
|
+
|
|
136
|
+
**When to use**:
|
|
137
|
+
- Updates 1-10 per second
|
|
138
|
+
- High concurrency (>10,000 clients)
|
|
139
|
+
- Near-instant updates required (<100ms latency)
|
|
140
|
+
|
|
141
|
+
**Architecture**:
|
|
142
|
+
```
|
|
143
|
+
┌─────────────────┐
|
|
144
|
+
│ Otto API │ ← Stateless HTTP (authentication, business logic)
|
|
145
|
+
│ (Puma) │
|
|
146
|
+
└─────────────────┘
|
|
147
|
+
↓
|
|
148
|
+
┌─────────┐
|
|
149
|
+
│ Redis │ ← Message queue (pub/sub)
|
|
150
|
+
│ Pub/Sub │
|
|
151
|
+
└─────────┘
|
|
152
|
+
↓
|
|
153
|
+
┌─────────────────┐
|
|
154
|
+
│ SSE Service │ ← Stateful streaming (Falcon/Iodine)
|
|
155
|
+
│ (Falcon) │
|
|
156
|
+
└─────────────────┘
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
**Example**: see "Otto + Falcon SSE Integration" in
|
|
160
|
+
`docs/1108-STREAMING_ARCHITECTURE_ANALYSIS.md`.
|
|
161
|
+
|
|
162
|
+
**Complexity**: ⭐⭐ Moderate
|
|
163
|
+
**Otto Integration**: ✅ Via Redis pub/sub
|
|
164
|
+
|
|
165
|
+
---
|
|
166
|
+
|
|
167
|
+
### Option 3: Third-Party Service (COMMERCIAL)
|
|
168
|
+
|
|
169
|
+
**When to use**:
|
|
170
|
+
- Don't want to manage WebSocket infrastructure
|
|
171
|
+
- Need global CDN distribution
|
|
172
|
+
- Require guaranteed SLA
|
|
173
|
+
|
|
174
|
+
**Options**:
|
|
175
|
+
- **Mercure**: Open-source SSE hub (self-hosted or managed)
|
|
176
|
+
- **Ably**: Commercial real-time messaging platform
|
|
177
|
+
- **Pusher**: Commercial WebSocket/SSE service
|
|
178
|
+
|
|
179
|
+
**Complexity**: ⭐⭐ Moderate (integration)
|
|
180
|
+
**Otto Integration**: ✅ Via HTTP API
|
|
181
|
+
|
|
182
|
+
---
|
|
183
|
+
|
|
184
|
+
## Documentation Created
|
|
185
|
+
|
|
186
|
+
### **docs/1108-STREAMING_ARCHITECTURE_ANALYSIS.md** (15,000+ words)
|
|
187
|
+
Comprehensive technical analysis covering:
|
|
188
|
+
- Otto's current architecture (detailed lifecycle analysis)
|
|
189
|
+
- Technical requirements for SSE/WebSocket (Rack 3, hijacking, etc.)
|
|
190
|
+
- Industry patterns (Rails, Sinatra, Roda, Go, Node.js)
|
|
191
|
+
- Compatibility analysis (why it doesn't fit)
|
|
192
|
+
- Best practices and anti-patterns
|
|
193
|
+
- Detailed recommendations with code examples
|
|
194
|
+
|
|
195
|
+
### Runnable examples: not yet published
|
|
196
|
+
|
|
197
|
+
Draft `examples/otto_falcon_sse_integration.rb` and
|
|
198
|
+
`examples/long_polling_example.rb` were written alongside this analysis but
|
|
199
|
+
targeted an Otto API surface that does not exist (`Otto::RequestContext`,
|
|
200
|
+
`enable_sessions!`, `raise_concern`, a `session` helper, `Rack::Handler::Puma`
|
|
201
|
+
under the Rack 3 pin) and used symbol-keyed params that Logic routes never
|
|
202
|
+
populate. They were withheld rather than shipped as copy-paste material; the
|
|
203
|
+
architecture they demonstrated is preserved in the snippets here and in the
|
|
204
|
+
analysis document.
|
|
205
|
+
|
|
206
|
+
---
|
|
207
|
+
|
|
208
|
+
## Key Insights
|
|
209
|
+
|
|
210
|
+
### The Real Question
|
|
211
|
+
|
|
212
|
+
**Not**: "Can Otto support SSE/WebSocket?"
|
|
213
|
+
(Technically possible with massive refactoring)
|
|
214
|
+
|
|
215
|
+
**But**: "Should Otto support SSE/WebSocket?"
|
|
216
|
+
(Architecturally inadvisable)
|
|
217
|
+
|
|
218
|
+
### Answer: **No**
|
|
219
|
+
|
|
220
|
+
Otto should remain focused on its core strengths:
|
|
221
|
+
- ✅ **Stateless** HTTP APIs
|
|
222
|
+
- ✅ **Security-first** design (CSRF, rate limiting, validation)
|
|
223
|
+
- ✅ **Privacy by default** (IP masking, geo-location)
|
|
224
|
+
- ✅ **Server-agnostic** (works with any Rack server)
|
|
225
|
+
- ✅ **Simple** and predictable architecture
|
|
226
|
+
|
|
227
|
+
Adding SSE/WebSocket would:
|
|
228
|
+
- ❌ Compromise architectural integrity
|
|
229
|
+
- ❌ Force specific async servers (Falcon, Iodine)
|
|
230
|
+
- ❌ Complicate security guarantees (middleware assumptions broken)
|
|
231
|
+
- ❌ Add significant complexity for niche use case
|
|
232
|
+
- ❌ Go against industry best practices (separation of concerns)
|
|
233
|
+
|
|
234
|
+
---
|
|
235
|
+
|
|
236
|
+
## What Otto SHOULD Do
|
|
237
|
+
|
|
238
|
+
### 1. ✅ Document Integration Patterns
|
|
239
|
+
|
|
240
|
+
Add official guide: "Integrating Otto with Real-Time Services"
|
|
241
|
+
- Long-polling patterns (built-in support)
|
|
242
|
+
- Separate SSE service pattern (Otto + Falcon + Redis)
|
|
243
|
+
- Third-party service integration (Mercure, Ably, Pusher)
|
|
244
|
+
|
|
245
|
+
### 2. ✅ Provide Example Code
|
|
246
|
+
|
|
247
|
+
Add to `examples/` directory, written against the real Logic-class contract and
|
|
248
|
+
verified to boot:
|
|
249
|
+
- `long_polling_example.rb`
|
|
250
|
+
- `otto_falcon_sse_integration.rb`
|
|
251
|
+
- `otto_mercure_integration.rb`
|
|
252
|
+
|
|
253
|
+
### 3. ⚠️ Consider Plugin System (If Community Demands)
|
|
254
|
+
|
|
255
|
+
**Only if there's strong demand**, create experimental plugin:
|
|
256
|
+
- Clearly marked "experimental" and "unsupported"
|
|
257
|
+
- Requires Falcon/Iodine (documented)
|
|
258
|
+
- Security implications documented
|
|
259
|
+
- No core changes required
|
|
260
|
+
|
|
261
|
+
### 4. ❌ Do NOT Add to Core
|
|
262
|
+
|
|
263
|
+
Preserve Otto's architectural integrity by:
|
|
264
|
+
- Keeping core stateless and synchronous
|
|
265
|
+
- Maintaining server-agnostic design
|
|
266
|
+
- Focusing on security and simplicity
|
|
267
|
+
- Following industry best practices (separation of concerns)
|
|
268
|
+
|
|
269
|
+
---
|
|
270
|
+
|
|
271
|
+
## Migration Guide for Existing Users
|
|
272
|
+
|
|
273
|
+
If you currently need real-time updates:
|
|
274
|
+
|
|
275
|
+
### Step 1: Assess Your Use Case
|
|
276
|
+
|
|
277
|
+
**Low-frequency updates (<1/min)**:
|
|
278
|
+
→ Use long-polling (Otto native support)
|
|
279
|
+
|
|
280
|
+
**Medium-frequency updates (1-10/sec)**:
|
|
281
|
+
→ Use separate SSE service (Otto + Falcon + Redis)
|
|
282
|
+
|
|
283
|
+
**High-frequency or bidirectional**:
|
|
284
|
+
→ Use separate WebSocket service or commercial solution
|
|
285
|
+
|
|
286
|
+
### Step 2: Implementation Path
|
|
287
|
+
|
|
288
|
+
#### For Long-Polling:
|
|
289
|
+
1. Create Otto route with long-polling logic
|
|
290
|
+
2. Use `sleep` loop with timeout
|
|
291
|
+
3. Client polls with timeout parameter
|
|
292
|
+
4. No external dependencies needed
|
|
293
|
+
|
|
294
|
+
See the `NotificationPollLogic` sketch above.
|
|
295
|
+
|
|
296
|
+
#### For Separate SSE Service:
|
|
297
|
+
1. Otto API handles authentication and publishes to Redis
|
|
298
|
+
2. Separate Falcon app subscribes to Redis and streams SSE
|
|
299
|
+
3. Client connects to SSE service with JWT token from Otto
|
|
300
|
+
4. Scale Otto and SSE services independently
|
|
301
|
+
|
|
302
|
+
See "Otto + Falcon SSE Integration" in
|
|
303
|
+
`docs/1108-STREAMING_ARCHITECTURE_ANALYSIS.md`.
|
|
304
|
+
|
|
305
|
+
### Step 3: Deployment
|
|
306
|
+
|
|
307
|
+
**Long-Polling**:
|
|
308
|
+
- Deploy with existing Otto setup (Puma, Unicorn, Passenger)
|
|
309
|
+
- Increase thread pool size for long-polling routes
|
|
310
|
+
- Monitor connection pool (ensure enough threads)
|
|
311
|
+
|
|
312
|
+
**Separate SSE Service**:
|
|
313
|
+
- Deploy Otto API with Puma (standard)
|
|
314
|
+
- Deploy Falcon SSE service separately (dedicated servers)
|
|
315
|
+
- Use Redis for pub/sub (cluster-ready)
|
|
316
|
+
- Configure Nginx with sticky sessions for SSE
|
|
317
|
+
- Scale services independently based on load
|
|
318
|
+
|
|
319
|
+
---
|
|
320
|
+
|
|
321
|
+
## Performance Guidance
|
|
322
|
+
|
|
323
|
+
### Long-Polling Capacity
|
|
324
|
+
|
|
325
|
+
**Example**: Puma with 5 workers × 32 threads = 160 concurrent requests
|
|
326
|
+
|
|
327
|
+
If long-polling uses 30s timeout:
|
|
328
|
+
- 160 concurrent connections
|
|
329
|
+
- ~320 clients with 50% utilization
|
|
330
|
+
- Up to 10,000 clients with proper thread tuning
|
|
331
|
+
|
|
332
|
+
**Good for**: Dashboard metrics, low-volume notifications
|
|
333
|
+
|
|
334
|
+
### SSE/WebSocket Capacity
|
|
335
|
+
|
|
336
|
+
**Example**: Falcon with 4 workers (async)
|
|
337
|
+
|
|
338
|
+
Each worker handles thousands of concurrent connections via fibers:
|
|
339
|
+
- 10,000+ concurrent SSE connections per server
|
|
340
|
+
- Horizontal scaling via Redis pub/sub
|
|
341
|
+
- Near-instant message delivery
|
|
342
|
+
|
|
343
|
+
**Good for**: Chat, multiplayer, high-frequency updates
|
|
344
|
+
|
|
345
|
+
---
|
|
346
|
+
|
|
347
|
+
## Conclusion
|
|
348
|
+
|
|
349
|
+
**Otto should NOT integrate SSE/WebSocket support** because:
|
|
350
|
+
|
|
351
|
+
1. **Architectural mismatch**: Stateless vs stateful paradigms
|
|
352
|
+
2. **Industry consensus**: Separate services is best practice
|
|
353
|
+
3. **Complexity cost**: Massive refactoring for niche use case
|
|
354
|
+
4. **Better alternatives**: Long-polling (simple) or separate services (powerful)
|
|
355
|
+
|
|
356
|
+
**Instead, Otto should**:
|
|
357
|
+
|
|
358
|
+
1. ✅ Document long-polling patterns (works today)
|
|
359
|
+
2. ✅ Provide integration examples (Otto + Falcon + Redis)
|
|
360
|
+
3. ✅ Recommend third-party solutions (Mercure, Ably, Pusher)
|
|
361
|
+
4. ✅ Stay focused on stateless HTTP APIs
|
|
362
|
+
|
|
363
|
+
**This preserves Otto's core strengths** while enabling users who need real-time functionality to integrate appropriate solutions.
|
|
364
|
+
|
|
365
|
+
---
|
|
366
|
+
|
|
367
|
+
## Further Reading
|
|
368
|
+
|
|
369
|
+
- **Technical Analysis**: `docs/1108-STREAMING_ARCHITECTURE_ANALYSIS.md`
|
|
370
|
+
- **Rack 3 Streaming**: https://github.com/rack/rack/issues/1600
|
|
371
|
+
- **ActionCable Architecture**: https://guides.rubyonrails.org/action_cable_overview.html
|
|
372
|
+
- **SSE vs WebSocket**: https://ably.com/blog/websockets-vs-sse
|
|
373
|
+
|
|
374
|
+
---
|
|
375
|
+
|
|
376
|
+
**End of Summary**
|
data/docs/geo-country.md
ADDED
|
@@ -0,0 +1,172 @@
|
|
|
1
|
+
# Geo-country resolution
|
|
2
|
+
|
|
3
|
+
Otto resolves a country-level ISO 3166-1 alpha-2 code for each request and
|
|
4
|
+
exposes it as `req.geo_country` / `env['otto.privacy.geo_country']`. Resolution
|
|
5
|
+
is country-only by design — that is the privacy posture; there is no city or
|
|
6
|
+
region lookup.
|
|
7
|
+
|
|
8
|
+
## Resolution order
|
|
9
|
+
|
|
10
|
+
`Otto::Privacy::GeoResolver.resolve` returns the first hit from:
|
|
11
|
+
|
|
12
|
+
1. **Application-configured header** (`geo_header:`) — e.g. `X-Client-Country`.
|
|
13
|
+
2. **Known provider headers** — Cloudflare (`CF-IPCountry`), AWS CloudFront,
|
|
14
|
+
Fastly, Akamai Edgescape, Azure Front Door, **Vercel**
|
|
15
|
+
(`X-Vercel-IP-Country`), and a few semi-standard names
|
|
16
|
+
(`X-Geo-Country`, `X-Country-Code`, `Country-Code`).
|
|
17
|
+
3. **Custom resolver** (`GeoResolver.custom_resolver`) — your own callable.
|
|
18
|
+
Unlike the other geo settings, this is **class-level** (see
|
|
19
|
+
[Configuration](#configuration)).
|
|
20
|
+
4. **Local MMDB database** (`geo_db_path:` / `geo_db_reader:`) — a MaxMind-DB
|
|
21
|
+
country database.
|
|
22
|
+
5. **`'**'`** — the unknown sentinel, when nothing else matches.
|
|
23
|
+
|
|
24
|
+
Steps 1 and 2 are **only consulted when geo headers can be trusted** (see
|
|
25
|
+
[Header trust](#header-trust-and-spoofing) below).
|
|
26
|
+
|
|
27
|
+
Resolution is **honest**: Otto does not guess from a hardcoded IP-range table.
|
|
28
|
+
When no header, custom resolver, or database resolves a country, the result is
|
|
29
|
+
`'**'`.
|
|
30
|
+
|
|
31
|
+
### Privacy: masked IP and masked env
|
|
32
|
+
|
|
33
|
+
The database lookup in step 4 runs on the request's **masked** IP
|
|
34
|
+
(e.g. `203.0.113.0`), never the real address. `check_geo_database` masks the IP
|
|
35
|
+
internally with the config's `octet_precision` before the lookup, so even a
|
|
36
|
+
direct `GeoResolver.resolve` caller passing a real IP does not expose it to the
|
|
37
|
+
database. Country-level MMDB networks are almost always ≥ /24, so the default
|
|
38
|
+
/24-masked value (`octet_precision: 1`) resolves to the same country.
|
|
39
|
+
|
|
40
|
+
In the middleware path Otto additionally hands `resolve` a **masked env view**:
|
|
41
|
+
`REMOTE_ADDR`, `X-Forwarded-For`, `X-Real-IP`, `X-Client-IP`, and the RFC 7239
|
|
42
|
+
`Forwarded` header are masked. So a `custom_resolver` cannot read the raw client
|
|
43
|
+
IP out of `env` either — use the `ip` argument (already masked), not `env`.
|
|
44
|
+
|
|
45
|
+
> **`octet_precision: 2`** masks two octets (a /16). That is coarser than most
|
|
46
|
+
> country networks, so it can reduce database hit rate for the small share of
|
|
47
|
+
> countries whose ranges are finer than /16 — those requests fall through to
|
|
48
|
+
> `'**'`. Header and custom-resolver sources are unaffected (they ignore the
|
|
49
|
+
> IP). Keep the default precision if you rely on the MMDB fallback.
|
|
50
|
+
|
|
51
|
+
## Configuration
|
|
52
|
+
|
|
53
|
+
All geo configuration is **boot-time only** (set once during single-threaded
|
|
54
|
+
initialization, before serving requests), matching `custom_resolver`'s
|
|
55
|
+
contract. `geo_header`, `geo_db_path`, and `geo_db_reader` are stored on the
|
|
56
|
+
instance's `Otto::Privacy::Config`, so separate Otto instances hold independent
|
|
57
|
+
geo configuration.
|
|
58
|
+
|
|
59
|
+
> **`custom_resolver` is the exception — it is class-level, not per-instance.**
|
|
60
|
+
> `GeoResolver.custom_resolver=` sets a singleton on the `GeoResolver` class, so
|
|
61
|
+
> it is **shared across every Otto instance in the process** (last write wins).
|
|
62
|
+
> If you run multiple Otto instances that need different resolver strategies,
|
|
63
|
+
> the custom resolver cannot distinguish them — branch inside a single resolver
|
|
64
|
+
> on `env`, or use per-instance `geo_db_reader` instead.
|
|
65
|
+
|
|
66
|
+
```ruby
|
|
67
|
+
otto.configure_ip_privacy(
|
|
68
|
+
geo: true, # default; false disables geo entirely
|
|
69
|
+
geo_header: 'X-Client-Country', # trusted app header (optional)
|
|
70
|
+
geo_db_path: 'data/geo-whois-asn-country.mmdb', # local MMDB fallback (optional)
|
|
71
|
+
# geo_db_reader: MaxMind::DB.new(path), # or bring your own reader (optional)
|
|
72
|
+
)
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
- **`geo: false`** short-circuits everything: no header reads, and any loaded
|
|
76
|
+
database is unloaded from memory (`req.geo_country` becomes `nil`).
|
|
77
|
+
- **`geo_header:`** accepts either the HTTP header name (`X-Client-Country`) or
|
|
78
|
+
the Rack CGI env key (`HTTP_X_CLIENT_COUNTRY`), in any case, and is
|
|
79
|
+
canonicalized to the env-key form. Pass `''` to clear.
|
|
80
|
+
- **`geo_db_path:`** is loaded once at boot in `MODE_MEMORY`. An unreadable
|
|
81
|
+
path, a corrupt/non-MMDB file, or a missing `maxmind-db` gem raises
|
|
82
|
+
`ArgumentError` **at configuration time**, not per-request. Pass `''` to
|
|
83
|
+
unload.
|
|
84
|
+
- **`geo_db_reader:`** injects any object responding to `#get(ip)` (a
|
|
85
|
+
preconfigured `MaxMind::DB` reader or a test double), keeping the reader and
|
|
86
|
+
data-source choice independent of Otto. It **overrides** `geo_db_path` when
|
|
87
|
+
both are given in the same call; supplying `geo_db_path` alone in a later call
|
|
88
|
+
clears a prior reader override.
|
|
89
|
+
|
|
90
|
+
Each keyword follows a `nil` = "leave unchanged" contract; pass `''` to a header
|
|
91
|
+
or path to clear it. Any geo-affecting change triggers the boot-time database
|
|
92
|
+
(re)load, so a bad `geo_db_path` fails at the `configure_ip_privacy` call.
|
|
93
|
+
|
|
94
|
+
## The database: gem and datafile
|
|
95
|
+
|
|
96
|
+
The reader and the data file are independent — the MMDB format is the interop
|
|
97
|
+
point.
|
|
98
|
+
|
|
99
|
+
### Reader gem (`maxmind-db`)
|
|
100
|
+
|
|
101
|
+
The [`maxmind-db`](https://rubygems.org/gems/maxmind-db) gem (official MaxMind
|
|
102
|
+
reader, Apache-2.0, pure Ruby, zero runtime deps) is an **optional**
|
|
103
|
+
dependency. Otto only `require`s it when a database is configured. Add it to
|
|
104
|
+
your app when you use the database fallback:
|
|
105
|
+
|
|
106
|
+
```ruby
|
|
107
|
+
# Gemfile
|
|
108
|
+
gem 'maxmind-db', '~> 1.4'
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
### Data file (`geo-whois-asn-country`)
|
|
112
|
+
|
|
113
|
+
The recommended data file is
|
|
114
|
+
[`geo-whois-asn-country`](https://github.com/sapics/ip-location-db) from
|
|
115
|
+
sapics/ip-location-db: **PDDL v1.0 (public domain, no attribution required)**,
|
|
116
|
+
rebuilt daily, shipped as MMDB. Otto vendors no database — country data goes
|
|
117
|
+
stale, and a public-domain file you refresh on your own schedule keeps
|
|
118
|
+
licensing and freshness in your control.
|
|
119
|
+
|
|
120
|
+
Download it (IPv4+IPv6) into a path of your choosing:
|
|
121
|
+
|
|
122
|
+
```bash
|
|
123
|
+
mkdir -p data
|
|
124
|
+
curl -fsSL -o data/geo-whois-asn-country.mmdb \
|
|
125
|
+
https://github.com/sapics/ip-location-db/releases/download/latest/geo-whois-asn-country.mmdb
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
Refresh it on your own schedule (e.g. a daily cron job running the same curl).
|
|
129
|
+
Any MMDB country database works — GeoLite2-Country, DB-IP Country Lite,
|
|
130
|
+
iplocate, etc. — since `GeoResolver` tolerates the record shapes country
|
|
131
|
+
databases actually use: nested `country.iso_code` (GeoLite2-Country style), a
|
|
132
|
+
flat `country_code` string, and a bare-string `country`.
|
|
133
|
+
|
|
134
|
+
> **Note on GeoLite2:** its EULA requires a MaxMind account/license key and
|
|
135
|
+
> obliges consumers to refresh within 30 days of each release. A PDDL dataset
|
|
136
|
+
> avoids both obligations.
|
|
137
|
+
|
|
138
|
+
## Header trust and spoofing
|
|
139
|
+
|
|
140
|
+
Every geo header is trivially client-spoofable unless the request actually
|
|
141
|
+
arrived through the CDN that sets it. Otto trusts geo headers — both the
|
|
142
|
+
configured `geo_header` and the provider headers — **only** for a request that
|
|
143
|
+
demonstrably arrived via a configured **CIDR trusted proxy**
|
|
144
|
+
(`env['otto.via_trusted_proxy']` with `trusted_proxies` configured). A spoofed
|
|
145
|
+
header on a direct connection is ignored, and resolution falls through to the
|
|
146
|
+
custom resolver / database.
|
|
147
|
+
|
|
148
|
+
Origins Otto cannot verify are **not** trusted:
|
|
149
|
+
|
|
150
|
+
- **No trusted-proxy configuration.** A direct internet client could otherwise
|
|
151
|
+
pick its own country by sending `CF-IPCountry` / `X-Client-Country`, so with
|
|
152
|
+
no `trusted_proxies` configured, header steps are skipped and resolution falls
|
|
153
|
+
to the resolver / database (`'**'` if neither is set).
|
|
154
|
+
- **Count-based `trusted_proxy_depth` mode.** The header-setting hop cannot be
|
|
155
|
+
verified as a geo-CDN, so depth mode does not enable header trust.
|
|
156
|
+
|
|
157
|
+
**Migration:** to keep header-based geo, configure `trusted_proxies` (CIDR
|
|
158
|
+
matchers) so Otto can verify the proxy origin. Depth-mode and header-only
|
|
159
|
+
deployments should set `geo_db_path` for a local database instead; otherwise
|
|
160
|
+
resolution returns `'**'`.
|
|
161
|
+
|
|
162
|
+
## Acceptance behavior summary
|
|
163
|
+
|
|
164
|
+
| Scenario | Result |
|
|
165
|
+
| --- | --- |
|
|
166
|
+
| Configured `geo_header` present and trusted | wins over provider headers |
|
|
167
|
+
| Request not via a verified CIDR trusted proxy | geo headers skipped |
|
|
168
|
+
| No `trusted_proxies` configured | geo headers skipped (not trusted) |
|
|
169
|
+
| Database lookup | uses the masked IP only |
|
|
170
|
+
| `geo: false` | `nil`, no database in memory |
|
|
171
|
+
| Bad `geo_db_path` | raises at boot, not per-request |
|
|
172
|
+
| Nothing matches | `'**'` |
|
|
@@ -177,12 +177,25 @@ deny). Everything fails closed.
|
|
|
177
177
|
This is the load-bearing decision, and it corrects the obvious-but-wrong first
|
|
178
178
|
instinct (which every initial design in the panel made).
|
|
179
179
|
|
|
180
|
-
`Otto::CaddyTLS::LocalhostGuard`
|
|
181
|
-
|
|
182
|
-
from forwarded headers.
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
180
|
+
`Otto::CaddyTLS::LocalhostGuard` authenticates the **original TCP socket peer**,
|
|
181
|
+
never the address left in `REMOTE_ADDR` after `IPPrivacyMiddleware` rewrites it
|
|
182
|
+
from forwarded headers.
|
|
183
|
+
|
|
184
|
+
Otto originally guaranteed that by ordering: the guard is installed via `Otto#use`
|
|
185
|
+
(appended, therefore *outermost* in Otto's `reduce`-built stack) and
|
|
186
|
+
`IPPrivacyMiddleware` was pinned *innermost*, so the guard ran first and read a
|
|
187
|
+
pristine `REMOTE_ADDR`. Issue #219 inverted that: masking innermost meant every
|
|
188
|
+
*other* middleware saw raw IPs, so `IPPrivacyMiddleware` is now pinned **outermost**
|
|
189
|
+
(the `:entrypoint` tier) and runs *ahead* of the guard.
|
|
190
|
+
|
|
191
|
+
The guarantee is preserved by a record rather than by order.
|
|
192
|
+
`IPPrivacyMiddleware` evaluates the untouched peer before masking and stores the
|
|
193
|
+
verdict as `env['otto.peer_loopback']` — a boolean, never an address, so it leaks
|
|
194
|
+
nothing. The guard reads that record when present and evaluates `REMOTE_ADDR`
|
|
195
|
+
itself when it is not (guard mounted outside Otto, or no privacy middleware in the
|
|
196
|
+
stack). Both paths share `Otto::Utils.loopback_address?`, so they cannot drift, and
|
|
197
|
+
the decision is made on the raw peer either way — regardless of when
|
|
198
|
+
`enable_caddy_tls!` is called.
|
|
186
199
|
|
|
187
200
|
Reading Otto's resolved `otto.client_ip` (or the rewritten `REMOTE_ADDR`) would be
|
|
188
201
|
**exploitable**: `Otto::Utils.resolve_client_ip` honors `X-Forwarded-For` when the
|
|
@@ -3,15 +3,46 @@
|
|
|
3
3
|
|
|
4
4
|
# Otto GeoResolver Extension Guide
|
|
5
5
|
#
|
|
6
|
-
#
|
|
7
|
-
#
|
|
8
|
-
#
|
|
6
|
+
# Otto resolves a country code in this order (first hit wins):
|
|
7
|
+
# 1. App-configured trusted header (configure_ip_privacy(geo_header:))
|
|
8
|
+
# 2. Built-in CDN/provider headers (Cloudflare, AWS, Vercel, ...)
|
|
9
|
+
# 3. Custom resolver hook (GeoResolver.custom_resolver = ...)
|
|
10
|
+
# 4. Local MMDB database (configure_ip_privacy(geo_db_path:/geo_db_reader:))
|
|
11
|
+
# 5. '**' (unknown) Otto does not guess from a hardcoded table
|
|
12
|
+
#
|
|
13
|
+
# This guide shows the extension points:
|
|
14
|
+
# A. Built-in configuration (trusted header + local database) — no code
|
|
15
|
+
# B. Custom resolver hook (inline or a callable object)
|
|
16
|
+
# C. Subclass-based (full control)
|
|
9
17
|
|
|
10
18
|
require 'bundler/setup'
|
|
11
19
|
require 'otto'
|
|
12
20
|
|
|
13
21
|
# =============================================================================
|
|
14
|
-
#
|
|
22
|
+
# A. Built-in configuration: trusted header + local country database
|
|
23
|
+
# =============================================================================
|
|
24
|
+
#
|
|
25
|
+
# No custom code needed — just configure the Otto instance. The database is
|
|
26
|
+
# looked up on the already-MASKED IP, and a bad geo_db_path fails at boot.
|
|
27
|
+
#
|
|
28
|
+
# otto = Otto.new('routes.txt')
|
|
29
|
+
# otto.configure_ip_privacy(
|
|
30
|
+
# geo_header: 'X-Client-Country', # trusted header checked before CDN headers
|
|
31
|
+
# geo_db_path: 'data/country.mmdb' # offline fallback (needs the maxmind-db gem)
|
|
32
|
+
# )
|
|
33
|
+
#
|
|
34
|
+
# Prefer to bring your own reader (any object responding to #get)? Inject it —
|
|
35
|
+
# this keeps the reader/data-source choice independent of Otto:
|
|
36
|
+
#
|
|
37
|
+
# reader = MaxMind::DB.new('data/country.mmdb', mode: MaxMind::DB::MODE_MEMORY)
|
|
38
|
+
# otto.configure_ip_privacy(geo_db_reader: reader)
|
|
39
|
+
#
|
|
40
|
+
# Security note: geo headers are only trusted for requests that arrive via a
|
|
41
|
+
# configured trusted proxy (add_trusted_proxy), since they are client-spoofable
|
|
42
|
+
# otherwise. configure_ip_privacy(geo: false) disables geo entirely.
|
|
43
|
+
|
|
44
|
+
# =============================================================================
|
|
45
|
+
# B. Quick Start: Custom resolver hook
|
|
15
46
|
# =============================================================================
|
|
16
47
|
|
|
17
48
|
puts 'Simple Custom Geo Resolution'
|
|
@@ -31,7 +62,9 @@ Otto::Privacy::GeoResolver.custom_resolver = custom_resolver
|
|
|
31
62
|
|
|
32
63
|
# Step 3: Test it
|
|
33
64
|
puts "1.2.3.4 -> #{Otto::Privacy::GeoResolver.resolve('1.2.3.4', {})}"
|
|
34
|
-
|
|
65
|
+
# Resolver returns nil for 8.8.8.8, and there is no header or database, so the
|
|
66
|
+
# honest answer is '**' (unknown) — Otto does not guess.
|
|
67
|
+
puts "8.8.8.8 -> #{Otto::Privacy::GeoResolver.resolve('8.8.8.8', {})} (unknown)"
|
|
35
68
|
|
|
36
69
|
# Reset for next example
|
|
37
70
|
Otto::Privacy::GeoResolver.custom_resolver = nil
|