otto 2.9.0 → 2.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.github/dependabot.yml +5 -0
- data/.github/workflows/ci.yml +11 -8
- data/.github/workflows/claude-code-review.yml +38 -13
- data/.github/workflows/claude.yml +10 -8
- data/.github/workflows/code-smells.yml +5 -5
- data/.github/workflows/release-gem.yml +2 -2
- data/.github/workflows/ruby-lint.yml +3 -3
- data/.github/workflows/yardoc.yml +5 -5
- data/.gitignore +1 -5
- data/.rubocop_todo.yml +7 -5
- data/AGENTS.md +22 -1
- data/CHANGELOG.rst +203 -0
- data/Gemfile +3 -3
- data/Gemfile.lock +11 -15
- data/README.md +75 -37
- data/docs/README.md +166 -0
- data/docs/adr/README.md +16 -0
- data/docs/adr/adr-001-route-authentication-at-handler-boundary.md +38 -0
- data/docs/adr/adr-002-multi-strategy-authentication-and-authorization.md +50 -0
- data/docs/adr/adr-003-caddy-tls-route-based-integration.md +48 -0
- data/docs/adr/adr-004-separate-compatibility-from-security-maintenance.md +58 -0
- data/docs/guides/authentication.md +377 -0
- data/docs/guides/caddy-tls.md +205 -0
- data/docs/guides/configuration_freezing.md +146 -0
- data/docs/guides/enrichment.md +161 -0
- data/docs/guides/forwarded-authority.md +249 -0
- data/docs/guides/geo-country.md +168 -0
- data/docs/guides/ip_privacy.md +39 -0
- data/docs/guides/ipaddr-encoding-quirk.md +56 -0
- data/docs/guides/mcp.md +282 -0
- data/docs/guides/privacy.md +193 -0
- data/docs/guides/routing.md +181 -0
- data/docs/guides/structured_logging.md +281 -0
- data/docs/guides/testing-guide.md +391 -0
- data/docs/maintainers/github-actions.md +41 -0
- data/docs/maintainers/investigations/.gitignore +2 -0
- data/docs/migrating/v2.0.0.md +337 -0
- data/docs/reference/authentication.md +290 -0
- data/docs/reference/route-syntax.md +181 -0
- data/docs/reference/runtime-and-dependency-security.md +86 -0
- data/examples/advanced_routes/README.md +43 -57
- data/examples/authentication_strategies/README.md +37 -196
- data/examples/basic/README.md +24 -39
- data/examples/basic/config.ru +0 -1
- data/examples/caddy_tls_demo/README.md +8 -2
- data/examples/lambda_handlers/README.md +11 -2
- data/examples/mcp_demo/README.md +75 -161
- data/examples/mcp_demo/config.ru +1 -0
- data/examples/security_features/README.md +36 -234
- data/lib/otto/caddy_tls/localhost_guard.rb +22 -24
- data/lib/otto/core/configuration.rb +36 -22
- data/lib/otto/core/file_safety.rb +89 -31
- data/lib/otto/core/middleware_stack.rb +36 -36
- data/lib/otto/core/router.rb +62 -19
- data/lib/otto/env_keys.rb +20 -2
- data/lib/otto/mcp/auth/token.rb +10 -4
- data/lib/otto/mcp/core.rb +23 -5
- data/lib/otto/mcp/endpoint.rb +41 -0
- data/lib/otto/mcp/errors.rb +15 -0
- data/lib/otto/mcp/options.rb +292 -0
- data/lib/otto/mcp/protocol.rb +52 -22
- data/lib/otto/mcp/rate_limiting.rb +175 -97
- data/lib/otto/mcp/registry.rb +14 -14
- data/lib/otto/mcp/schema_validation.rb +20 -12
- data/lib/otto/mcp/server.rb +131 -32
- data/lib/otto/optional_dependency.rb +57 -0
- data/lib/otto/privacy/config.rb +10 -8
- data/lib/otto/security/authentication/route_auth_wrapper/role_authorization.rb +22 -5
- data/lib/otto/security/authentication/strategies/api_key_strategy.rb +181 -18
- data/lib/otto/security/authentication/strategies/permission_strategy.rb +1 -0
- data/lib/otto/security/authentication/strategies/role_strategy.rb +1 -0
- data/lib/otto/security/authentication/strategies/session_strategy.rb +1 -0
- data/lib/otto/security/authentication/strategy_result.rb +58 -28
- data/lib/otto/security/config.rb +328 -14
- data/lib/otto/security/configurator.rb +36 -6
- data/lib/otto/security/core.rb +19 -1
- data/lib/otto/security/middleware/ip_privacy_middleware.rb +105 -4
- data/lib/otto/security/middleware/rate_limit_middleware.rb +1 -6
- data/lib/otto/security/rate_limiter.rb +81 -46
- data/lib/otto/utils.rb +50 -0
- data/lib/otto/version.rb +1 -1
- data/lib/otto.rb +9 -11
- data/otto.gemspec +0 -2
- metadata +31 -41
- data/docs/.gitignore +0 -10
- data/docs/1108-STREAMING_ARCHITECTURE_ANALYSIS.md +0 -1105
- data/docs/1108-STREAMING_SUPPORT_SUMMARY.md +0 -376
- data/docs/enrichment.md +0 -128
- data/docs/geo-country.md +0 -181
- data/docs/ipaddr-encoding-quirk.md +0 -34
- data/docs/migrating/v2.0.0-pre1.md +0 -276
- data/docs/migrating/v2.0.0-pre2.md +0 -338
- data/docs/modern-authentication-authorization-landscape.md +0 -558
- data/docs/multi-strategy-authentication-design.md +0 -1401
- data/docs/reverse-proxy-network-services.md +0 -371
|
@@ -1,376 +0,0 @@
|
|
|
1
|
-
# Otto Streaming Support: Executive Summary
|
|
2
|
-
|
|
3
|
-
**Investigation Date**: 2025-11-08
|
|
4
|
-
**Question**: Should Otto support Server-Sent Events (SSE) and WebSockets?
|
|
5
|
-
**Answer**: **No** - Use separate services or long-polling instead
|
|
6
|
-
|
|
7
|
-
> **Note**: The Ruby snippets below illustrate the recommended architecture. They
|
|
8
|
-
> are written against the current Otto Logic-class contract
|
|
9
|
-
> (`initialize(context, params, locale)` + `process`), but have not been executed
|
|
10
|
-
> end-to-end. Runnable examples are tracked separately.
|
|
11
|
-
|
|
12
|
-
---
|
|
13
|
-
|
|
14
|
-
## Quick Recommendation
|
|
15
|
-
|
|
16
|
-
| Use Case | Solution | Complexity | Otto Compatible? |
|
|
17
|
-
|----------|----------|------------|------------------|
|
|
18
|
-
| **Low-frequency updates (<1/min)** | Long-polling | ⭐ Simple | ✅ Yes |
|
|
19
|
-
| **Medium-frequency updates (1-10/sec)** | Separate SSE service | ⭐⭐ Moderate | ✅ Via integration |
|
|
20
|
-
| **High-frequency updates (>10/sec)** | Separate WebSocket service | ⭐⭐⭐ Complex | ✅ Via integration |
|
|
21
|
-
| **Bidirectional communication** | Separate WebSocket service | ⭐⭐⭐ Complex | ✅ Via integration |
|
|
22
|
-
|
|
23
|
-
---
|
|
24
|
-
|
|
25
|
-
## Key Findings
|
|
26
|
-
|
|
27
|
-
### 1. **Otto's Architecture is Fundamentally Incompatible with Streaming**
|
|
28
|
-
|
|
29
|
-
Otto is designed as a **stateless, synchronous, request/response** framework:
|
|
30
|
-
|
|
31
|
-
```
|
|
32
|
-
Request → Middleware → Route → Handler → Response → Close Connection
|
|
33
|
-
```
|
|
34
|
-
|
|
35
|
-
SSE/WebSocket require **stateful, long-lived connections**:
|
|
36
|
-
|
|
37
|
-
```
|
|
38
|
-
Request → Upgrade → Keep Open → Stream Data (minutes/hours) → Close
|
|
39
|
-
```
|
|
40
|
-
|
|
41
|
-
**Incompatibilities**:
|
|
42
|
-
- ❌ Response handlers expect complete responses (not streaming enumerators)
|
|
43
|
-
- ❌ Middleware stack can't unwind during long-lived connections
|
|
44
|
-
- ❌ Configuration freezing prevents runtime streaming adjustments
|
|
45
|
-
- ❌ Requires async servers (Falcon, Iodine) - breaks server-agnostic design
|
|
46
|
-
- ❌ Stateful routing complicates horizontal scaling
|
|
47
|
-
|
|
48
|
-
### 2. **Industry Best Practice: Separate Services**
|
|
49
|
-
|
|
50
|
-
Modern frameworks separate real-time communication from REST APIs:
|
|
51
|
-
|
|
52
|
-
**Rails (ActionCable)**:
|
|
53
|
-
```
|
|
54
|
-
Rails App (Puma) → Redis Pub/Sub ← ActionCable (Separate Process)
|
|
55
|
-
```
|
|
56
|
-
|
|
57
|
-
**Node.js (Express + Socket.IO)**:
|
|
58
|
-
```
|
|
59
|
-
Express (HTTP Routes) + Socket.IO (Separate Layer)
|
|
60
|
-
```
|
|
61
|
-
|
|
62
|
-
**Benefits**:
|
|
63
|
-
- ✅ Independent scaling (scale WebSocket separately from API)
|
|
64
|
-
- ✅ Technology choice (use best tool for each job)
|
|
65
|
-
- ✅ Fault isolation (WebSocket crash doesn't affect API)
|
|
66
|
-
- ✅ Clear architectural boundaries
|
|
67
|
-
|
|
68
|
-
### 3. **Long-Polling Works Perfectly with Otto**
|
|
69
|
-
|
|
70
|
-
For low-to-medium frequency updates, long-polling is **simple and effective**:
|
|
71
|
-
|
|
72
|
-
```ruby
|
|
73
|
-
# Otto route (works with any Rack server)
|
|
74
|
-
GET /api/notifications/poll NotificationPollLogic response=json auth=session
|
|
75
|
-
|
|
76
|
-
class NotificationPollLogic
|
|
77
|
-
attr_reader :context, :params, :locale
|
|
78
|
-
|
|
79
|
-
def initialize(context, params, locale)
|
|
80
|
-
@context = context
|
|
81
|
-
@params = params
|
|
82
|
-
@locale = locale
|
|
83
|
-
end
|
|
84
|
-
|
|
85
|
-
def process
|
|
86
|
-
# Logic-class params are string-keyed (LogicClassHandler does not apply
|
|
87
|
-
# Otto::Static.indifferent_params).
|
|
88
|
-
timeout = params['timeout'].to_i.clamp(1, 30)
|
|
89
|
-
last_id = params['last_id'].to_i
|
|
90
|
-
start_time = Time.now
|
|
91
|
-
|
|
92
|
-
# Identity comes from the StrategyResult, never from request params.
|
|
93
|
-
user_id = context.user_id
|
|
94
|
-
|
|
95
|
-
loop do
|
|
96
|
-
notifications = fetch_new(user_id, last_id)
|
|
97
|
-
return { notifications: notifications } if notifications.any?
|
|
98
|
-
|
|
99
|
-
break if Time.now - start_time > timeout
|
|
100
|
-
|
|
101
|
-
sleep 0.5
|
|
102
|
-
end
|
|
103
|
-
|
|
104
|
-
{ notifications: [] }
|
|
105
|
-
end
|
|
106
|
-
end
|
|
107
|
-
```
|
|
108
|
-
|
|
109
|
-
**Benefits**:
|
|
110
|
-
- ✅ HTTP-based (cacheable, proxy-friendly, standard tooling)
|
|
111
|
-
- ✅ Works with Otto's synchronous model
|
|
112
|
-
- ✅ Compatible with any Rack server (Puma, Unicorn, Passenger)
|
|
113
|
-
- ✅ Simple debugging (standard HTTP requests/responses)
|
|
114
|
-
- ✅ No external dependencies
|
|
115
|
-
|
|
116
|
-
---
|
|
117
|
-
|
|
118
|
-
## Recommended Solutions
|
|
119
|
-
|
|
120
|
-
### Option 1: Long-Polling (SIMPLEST)
|
|
121
|
-
|
|
122
|
-
**When to use**:
|
|
123
|
-
- Updates less than 1 per minute
|
|
124
|
-
- Moderate concurrency (<10,000 clients)
|
|
125
|
-
- Simple deployment preferred
|
|
126
|
-
|
|
127
|
-
**Example**: see the `NotificationPollLogic` sketch above.
|
|
128
|
-
|
|
129
|
-
**Complexity**: ⭐ Simple
|
|
130
|
-
**Otto Integration**: ✅ Native support (no changes needed)
|
|
131
|
-
|
|
132
|
-
---
|
|
133
|
-
|
|
134
|
-
### Option 2: Separate SSE Service (RECOMMENDED FOR REAL-TIME)
|
|
135
|
-
|
|
136
|
-
**When to use**:
|
|
137
|
-
- Updates 1-10 per second
|
|
138
|
-
- High concurrency (>10,000 clients)
|
|
139
|
-
- Near-instant updates required (<100ms latency)
|
|
140
|
-
|
|
141
|
-
**Architecture**:
|
|
142
|
-
```
|
|
143
|
-
┌─────────────────┐
|
|
144
|
-
│ Otto API │ ← Stateless HTTP (authentication, business logic)
|
|
145
|
-
│ (Puma) │
|
|
146
|
-
└─────────────────┘
|
|
147
|
-
↓
|
|
148
|
-
┌─────────┐
|
|
149
|
-
│ Redis │ ← Message queue (pub/sub)
|
|
150
|
-
│ Pub/Sub │
|
|
151
|
-
└─────────┘
|
|
152
|
-
↓
|
|
153
|
-
┌─────────────────┐
|
|
154
|
-
│ SSE Service │ ← Stateful streaming (Falcon/Iodine)
|
|
155
|
-
│ (Falcon) │
|
|
156
|
-
└─────────────────┘
|
|
157
|
-
```
|
|
158
|
-
|
|
159
|
-
**Example**: see "Otto + Falcon SSE Integration" in
|
|
160
|
-
`docs/1108-STREAMING_ARCHITECTURE_ANALYSIS.md`.
|
|
161
|
-
|
|
162
|
-
**Complexity**: ⭐⭐ Moderate
|
|
163
|
-
**Otto Integration**: ✅ Via Redis pub/sub
|
|
164
|
-
|
|
165
|
-
---
|
|
166
|
-
|
|
167
|
-
### Option 3: Third-Party Service (COMMERCIAL)
|
|
168
|
-
|
|
169
|
-
**When to use**:
|
|
170
|
-
- Don't want to manage WebSocket infrastructure
|
|
171
|
-
- Need global CDN distribution
|
|
172
|
-
- Require guaranteed SLA
|
|
173
|
-
|
|
174
|
-
**Options**:
|
|
175
|
-
- **Mercure**: Open-source SSE hub (self-hosted or managed)
|
|
176
|
-
- **Ably**: Commercial real-time messaging platform
|
|
177
|
-
- **Pusher**: Commercial WebSocket/SSE service
|
|
178
|
-
|
|
179
|
-
**Complexity**: ⭐⭐ Moderate (integration)
|
|
180
|
-
**Otto Integration**: ✅ Via HTTP API
|
|
181
|
-
|
|
182
|
-
---
|
|
183
|
-
|
|
184
|
-
## Documentation Created
|
|
185
|
-
|
|
186
|
-
### **docs/1108-STREAMING_ARCHITECTURE_ANALYSIS.md** (15,000+ words)
|
|
187
|
-
Comprehensive technical analysis covering:
|
|
188
|
-
- Otto's current architecture (detailed lifecycle analysis)
|
|
189
|
-
- Technical requirements for SSE/WebSocket (Rack 3, hijacking, etc.)
|
|
190
|
-
- Industry patterns (Rails, Sinatra, Roda, Go, Node.js)
|
|
191
|
-
- Compatibility analysis (why it doesn't fit)
|
|
192
|
-
- Best practices and anti-patterns
|
|
193
|
-
- Detailed recommendations with code examples
|
|
194
|
-
|
|
195
|
-
### Runnable examples: not yet published
|
|
196
|
-
|
|
197
|
-
Draft `examples/otto_falcon_sse_integration.rb` and
|
|
198
|
-
`examples/long_polling_example.rb` were written alongside this analysis but
|
|
199
|
-
targeted an Otto API surface that does not exist (`Otto::RequestContext`,
|
|
200
|
-
`enable_sessions!`, `raise_concern`, a `session` helper, `Rack::Handler::Puma`
|
|
201
|
-
under the Rack 3 pin) and used symbol-keyed params that Logic routes never
|
|
202
|
-
populate. They were withheld rather than shipped as copy-paste material; the
|
|
203
|
-
architecture they demonstrated is preserved in the snippets here and in the
|
|
204
|
-
analysis document.
|
|
205
|
-
|
|
206
|
-
---
|
|
207
|
-
|
|
208
|
-
## Key Insights
|
|
209
|
-
|
|
210
|
-
### The Real Question
|
|
211
|
-
|
|
212
|
-
**Not**: "Can Otto support SSE/WebSocket?"
|
|
213
|
-
(Technically possible with massive refactoring)
|
|
214
|
-
|
|
215
|
-
**But**: "Should Otto support SSE/WebSocket?"
|
|
216
|
-
(Architecturally inadvisable)
|
|
217
|
-
|
|
218
|
-
### Answer: **No**
|
|
219
|
-
|
|
220
|
-
Otto should remain focused on its core strengths:
|
|
221
|
-
- ✅ **Stateless** HTTP APIs
|
|
222
|
-
- ✅ **Security-first** design (CSRF, rate limiting, validation)
|
|
223
|
-
- ✅ **Privacy by default** (IP masking, geo-location)
|
|
224
|
-
- ✅ **Server-agnostic** (works with any Rack server)
|
|
225
|
-
- ✅ **Simple** and predictable architecture
|
|
226
|
-
|
|
227
|
-
Adding SSE/WebSocket would:
|
|
228
|
-
- ❌ Compromise architectural integrity
|
|
229
|
-
- ❌ Force specific async servers (Falcon, Iodine)
|
|
230
|
-
- ❌ Complicate security guarantees (middleware assumptions broken)
|
|
231
|
-
- ❌ Add significant complexity for niche use case
|
|
232
|
-
- ❌ Go against industry best practices (separation of concerns)
|
|
233
|
-
|
|
234
|
-
---
|
|
235
|
-
|
|
236
|
-
## What Otto SHOULD Do
|
|
237
|
-
|
|
238
|
-
### 1. ✅ Document Integration Patterns
|
|
239
|
-
|
|
240
|
-
Add official guide: "Integrating Otto with Real-Time Services"
|
|
241
|
-
- Long-polling patterns (built-in support)
|
|
242
|
-
- Separate SSE service pattern (Otto + Falcon + Redis)
|
|
243
|
-
- Third-party service integration (Mercure, Ably, Pusher)
|
|
244
|
-
|
|
245
|
-
### 2. ✅ Provide Example Code
|
|
246
|
-
|
|
247
|
-
Add to `examples/` directory, written against the real Logic-class contract and
|
|
248
|
-
verified to boot:
|
|
249
|
-
- `long_polling_example.rb`
|
|
250
|
-
- `otto_falcon_sse_integration.rb`
|
|
251
|
-
- `otto_mercure_integration.rb`
|
|
252
|
-
|
|
253
|
-
### 3. ⚠️ Consider Plugin System (If Community Demands)
|
|
254
|
-
|
|
255
|
-
**Only if there's strong demand**, create experimental plugin:
|
|
256
|
-
- Clearly marked "experimental" and "unsupported"
|
|
257
|
-
- Requires Falcon/Iodine (documented)
|
|
258
|
-
- Security implications documented
|
|
259
|
-
- No core changes required
|
|
260
|
-
|
|
261
|
-
### 4. ❌ Do NOT Add to Core
|
|
262
|
-
|
|
263
|
-
Preserve Otto's architectural integrity by:
|
|
264
|
-
- Keeping core stateless and synchronous
|
|
265
|
-
- Maintaining server-agnostic design
|
|
266
|
-
- Focusing on security and simplicity
|
|
267
|
-
- Following industry best practices (separation of concerns)
|
|
268
|
-
|
|
269
|
-
---
|
|
270
|
-
|
|
271
|
-
## Migration Guide for Existing Users
|
|
272
|
-
|
|
273
|
-
If you currently need real-time updates:
|
|
274
|
-
|
|
275
|
-
### Step 1: Assess Your Use Case
|
|
276
|
-
|
|
277
|
-
**Low-frequency updates (<1/min)**:
|
|
278
|
-
→ Use long-polling (Otto native support)
|
|
279
|
-
|
|
280
|
-
**Medium-frequency updates (1-10/sec)**:
|
|
281
|
-
→ Use separate SSE service (Otto + Falcon + Redis)
|
|
282
|
-
|
|
283
|
-
**High-frequency or bidirectional**:
|
|
284
|
-
→ Use separate WebSocket service or commercial solution
|
|
285
|
-
|
|
286
|
-
### Step 2: Implementation Path
|
|
287
|
-
|
|
288
|
-
#### For Long-Polling:
|
|
289
|
-
1. Create Otto route with long-polling logic
|
|
290
|
-
2. Use `sleep` loop with timeout
|
|
291
|
-
3. Client polls with timeout parameter
|
|
292
|
-
4. No external dependencies needed
|
|
293
|
-
|
|
294
|
-
See the `NotificationPollLogic` sketch above.
|
|
295
|
-
|
|
296
|
-
#### For Separate SSE Service:
|
|
297
|
-
1. Otto API handles authentication and publishes to Redis
|
|
298
|
-
2. Separate Falcon app subscribes to Redis and streams SSE
|
|
299
|
-
3. Client connects to SSE service with JWT token from Otto
|
|
300
|
-
4. Scale Otto and SSE services independently
|
|
301
|
-
|
|
302
|
-
See "Otto + Falcon SSE Integration" in
|
|
303
|
-
`docs/1108-STREAMING_ARCHITECTURE_ANALYSIS.md`.
|
|
304
|
-
|
|
305
|
-
### Step 3: Deployment
|
|
306
|
-
|
|
307
|
-
**Long-Polling**:
|
|
308
|
-
- Deploy with existing Otto setup (Puma, Unicorn, Passenger)
|
|
309
|
-
- Increase thread pool size for long-polling routes
|
|
310
|
-
- Monitor connection pool (ensure enough threads)
|
|
311
|
-
|
|
312
|
-
**Separate SSE Service**:
|
|
313
|
-
- Deploy Otto API with Puma (standard)
|
|
314
|
-
- Deploy Falcon SSE service separately (dedicated servers)
|
|
315
|
-
- Use Redis for pub/sub (cluster-ready)
|
|
316
|
-
- Configure Nginx with sticky sessions for SSE
|
|
317
|
-
- Scale services independently based on load
|
|
318
|
-
|
|
319
|
-
---
|
|
320
|
-
|
|
321
|
-
## Performance Guidance
|
|
322
|
-
|
|
323
|
-
### Long-Polling Capacity
|
|
324
|
-
|
|
325
|
-
**Example**: Puma with 5 workers × 32 threads = 160 concurrent requests
|
|
326
|
-
|
|
327
|
-
If long-polling uses 30s timeout:
|
|
328
|
-
- 160 concurrent connections
|
|
329
|
-
- ~320 clients with 50% utilization
|
|
330
|
-
- Up to 10,000 clients with proper thread tuning
|
|
331
|
-
|
|
332
|
-
**Good for**: Dashboard metrics, low-volume notifications
|
|
333
|
-
|
|
334
|
-
### SSE/WebSocket Capacity
|
|
335
|
-
|
|
336
|
-
**Example**: Falcon with 4 workers (async)
|
|
337
|
-
|
|
338
|
-
Each worker handles thousands of concurrent connections via fibers:
|
|
339
|
-
- 10,000+ concurrent SSE connections per server
|
|
340
|
-
- Horizontal scaling via Redis pub/sub
|
|
341
|
-
- Near-instant message delivery
|
|
342
|
-
|
|
343
|
-
**Good for**: Chat, multiplayer, high-frequency updates
|
|
344
|
-
|
|
345
|
-
---
|
|
346
|
-
|
|
347
|
-
## Conclusion
|
|
348
|
-
|
|
349
|
-
**Otto should NOT integrate SSE/WebSocket support** because:
|
|
350
|
-
|
|
351
|
-
1. **Architectural mismatch**: Stateless vs stateful paradigms
|
|
352
|
-
2. **Industry consensus**: Separate services is best practice
|
|
353
|
-
3. **Complexity cost**: Massive refactoring for niche use case
|
|
354
|
-
4. **Better alternatives**: Long-polling (simple) or separate services (powerful)
|
|
355
|
-
|
|
356
|
-
**Instead, Otto should**:
|
|
357
|
-
|
|
358
|
-
1. ✅ Document long-polling patterns (works today)
|
|
359
|
-
2. ✅ Provide integration examples (Otto + Falcon + Redis)
|
|
360
|
-
3. ✅ Recommend third-party solutions (Mercure, Ably, Pusher)
|
|
361
|
-
4. ✅ Stay focused on stateless HTTP APIs
|
|
362
|
-
|
|
363
|
-
**This preserves Otto's core strengths** while enabling users who need real-time functionality to integrate appropriate solutions.
|
|
364
|
-
|
|
365
|
-
---
|
|
366
|
-
|
|
367
|
-
## Further Reading
|
|
368
|
-
|
|
369
|
-
- **Technical Analysis**: `docs/1108-STREAMING_ARCHITECTURE_ANALYSIS.md`
|
|
370
|
-
- **Rack 3 Streaming**: https://github.com/rack/rack/issues/1600
|
|
371
|
-
- **ActionCable Architecture**: https://guides.rubyonrails.org/action_cable_overview.html
|
|
372
|
-
- **SSE vs WebSocket**: https://ably.com/blog/websockets-vs-sse
|
|
373
|
-
|
|
374
|
-
---
|
|
375
|
-
|
|
376
|
-
**End of Summary**
|
data/docs/enrichment.md
DELETED
|
@@ -1,128 +0,0 @@
|
|
|
1
|
-
# ASN and anonymizer enrichment
|
|
2
|
-
|
|
3
|
-
Two opt-in, country-adjacent signals layered on the same privacy pipeline as
|
|
4
|
-
[geo-country](geo-country.md):
|
|
5
|
-
|
|
6
|
-
- **ASN** — the network operator an address belongs to, as `req.asn` /
|
|
7
|
-
`env['otto.privacy.asn']`: `'AS15169'`.
|
|
8
|
-
- **Anonymizer** — whether the address is a known anonymizing egress, as
|
|
9
|
-
`req.anonymizer` / `env['otto.privacy.anonymizer']`: `'tor'`, `'proxy'`,
|
|
10
|
-
`'vpn'`, `'residential_proxy'`, `'hosting'`, `'anonymous'`, `'none'`, `'**'`.
|
|
11
|
-
|
|
12
|
-
Both are **off by default** (unlike geo) and **database-only**: no CDN
|
|
13
|
-
publishes a client-ASN or anonymizer header with meaningful deployment, so
|
|
14
|
-
there is no header tier and none of geo's header-trust machinery applies —
|
|
15
|
-
including the `geo_header`/`trusted_proxy_depth` boot conflict.
|
|
16
|
-
|
|
17
|
-
Every signal keeps the same three-state contract:
|
|
18
|
-
|
|
19
|
-
| Value | Meaning |
|
|
20
|
-
| --- | --- |
|
|
21
|
-
| `nil` | the signal is switched off |
|
|
22
|
-
| `'**'` | switched on, but nothing resolved (no database, or lookup failed) |
|
|
23
|
-
| a label | a real answer |
|
|
24
|
-
|
|
25
|
-
## Configuration
|
|
26
|
-
|
|
27
|
-
```ruby
|
|
28
|
-
otto.configure_ip_privacy(
|
|
29
|
-
asn: true,
|
|
30
|
-
asn_db_path: 'data/origin-asn.mmdb',
|
|
31
|
-
anonymizer: true,
|
|
32
|
-
anonymizer_db_path: 'data/anonymizer.mmdb',
|
|
33
|
-
)
|
|
34
|
-
```
|
|
35
|
-
|
|
36
|
-
Both accept the same bring-your-own-reader seam as geo (`asn_db_reader:` /
|
|
37
|
-
`anonymizer_db_reader:` — any object responding to `#get(ip)`; a reader
|
|
38
|
-
supplied in the same call wins over a path). Bad paths raise at boot, not
|
|
39
|
-
per-request. The [`maxmind-db`](https://rubygems.org/gems/maxmind-db) gem is
|
|
40
|
-
required only when a `*_db_path` is configured.
|
|
41
|
-
|
|
42
|
-
## ASN: which address is looked up, and why that's safe
|
|
43
|
-
|
|
44
|
-
The ASN lookup uses the **masked** IP, same as geo. This is not a compromise:
|
|
45
|
-
IPv4 BGP routes are not announced longer than /24, so a /24-masked address
|
|
46
|
-
falls inside the same announced prefix — and therefore the same ASN — as the
|
|
47
|
-
real one. (IPv6 is coarser: at `octet_precision: 1` Otto zeroes the last 80
|
|
48
|
-
bits, wider than many IPv6 announcements, so treat IPv6 ASN as best-effort.)
|
|
49
|
-
|
|
50
|
-
### Data file
|
|
51
|
-
|
|
52
|
-
> **Naming caution:** the country database this project recommends,
|
|
53
|
-
> `geo-whois-asn-country`, does **not** contain ASN data. In
|
|
54
|
-
> sapics/ip-location-db naming, `geo-whois-asn` describes the *sources* the
|
|
55
|
-
> country data was derived from; the final token (`-country`) is what the
|
|
56
|
-
> records contain. ASN data is a separate file type in that project.
|
|
57
|
-
|
|
58
|
-
The recommended ASN file is
|
|
59
|
-
[`origin-asn`](https://github.com/sapics/ip-location-db/tree/main/origin-asn/)
|
|
60
|
-
from sapics/ip-location-db — like `geo-whois-asn-country` it is **PDDL v1.0
|
|
61
|
-
(public domain)** and rebuilt daily, so the licensing/freshness posture
|
|
62
|
-
matches the geo guidance:
|
|
63
|
-
|
|
64
|
-
```bash
|
|
65
|
-
curl -fsSL -o data/origin-asn.mmdb \
|
|
66
|
-
https://github.com/sapics/ip-location-db/releases/download/latest/origin-asn.mmdb
|
|
67
|
-
```
|
|
68
|
-
|
|
69
|
-
Its records carry a flat `autonomous_system_number` (verified against the
|
|
70
|
-
published file), which is the primary key `AsnResolver` reads. GeoLite2-ASN
|
|
71
|
-
and DB-IP ASN Lite MMDBs work too (same key; GeoLite2's EULA caveats from the
|
|
72
|
-
geo doc apply). Reserved ASNs (0 per RFC 7607, the AS_TRANS placeholder 23456
|
|
73
|
-
per RFC 6793) resolve to `'**'` rather than being reported as operators.
|
|
74
|
-
|
|
75
|
-
## Anonymizer: the one unmasked lookup
|
|
76
|
-
|
|
77
|
-
Anonymizer classification reads the **unmasked** address. This is deliberate
|
|
78
|
-
and documented in `AnonymizerResolver` itself: anonymizer databases list
|
|
79
|
-
individual egress nodes at or near /32, so the /24 equivalence that justifies
|
|
80
|
-
masked geo and ASN lookups does not hold. A masked lookup would flag a whole
|
|
81
|
-
/24 because one host in it runs a Tor exit, and miss the exit node itself —
|
|
82
|
-
wrong in both directions.
|
|
83
|
-
|
|
84
|
-
The privacy containment is the same one Otto already relies on for
|
|
85
|
-
`hash_ip` and `env['otto.ip_match']`, which also consume the full IP: **only
|
|
86
|
-
the derived value leaves**. The resolver returns a label; the address is
|
|
87
|
-
never persisted, serialized, or handed downstream.
|
|
88
|
-
|
|
89
|
-
### Reading the labels
|
|
90
|
-
|
|
91
|
-
When several database flags are set at once (a Tor exit hosted at a cloud
|
|
92
|
-
provider), the **most specific** label wins: `tor` > `proxy` > `vpn` >
|
|
93
|
-
`residential_proxy` > `hosting` > `anonymous`.
|
|
94
|
-
|
|
95
|
-
Two labels deserve care:
|
|
96
|
-
|
|
97
|
-
- **`'none'`** means the database was consulted and does not list the
|
|
98
|
-
address. For an anonymizer database that is a real answer (these files
|
|
99
|
-
record only flagged addresses), but it is *not* a positive assertion the
|
|
100
|
-
visitor is residential — it is only as fresh as your database file.
|
|
101
|
-
- **`'**'`** means no database answered at all. Do not collapse it into
|
|
102
|
-
`'none'`: `'none'` is evidence, `'**'` is the absence of evidence. A
|
|
103
|
-
"block anonymizers" rule that treats `'**'` as `'none'` fails open when
|
|
104
|
-
the database file goes missing.
|
|
105
|
-
|
|
106
|
-
### Data file
|
|
107
|
-
|
|
108
|
-
There is no public-domain anonymizer dataset of `origin-asn`'s quality; this
|
|
109
|
-
signal is bring-your-own-database. `AnonymizerResolver` reads the MaxMind
|
|
110
|
-
GeoIP2 Anonymous-IP flag schema (`is_tor_exit_node`, `is_public_proxy`,
|
|
111
|
-
`is_anonymous_vpn`, `is_residential_proxy`, `is_hosting_provider`,
|
|
112
|
-
`is_anonymous`), which commercial and self-built MMDBs alike use. A workable
|
|
113
|
-
self-built option: compile the [Tor bulk exit
|
|
114
|
-
list](https://check.torproject.org/torbulkexitlist) into an MMDB with
|
|
115
|
-
`is_tor_exit_node` set — that covers the highest-signal label with fully
|
|
116
|
-
public data.
|
|
117
|
-
|
|
118
|
-
## Acceptance behavior summary
|
|
119
|
-
|
|
120
|
-
| Scenario | Result |
|
|
121
|
-
| --- | --- |
|
|
122
|
-
| Signal not enabled | `nil` everywhere (env key absent for exempt IPs) |
|
|
123
|
-
| Enabled, no database configured | `'**'` |
|
|
124
|
-
| Database read raises | `'**'` (a lookup must never crash a request) |
|
|
125
|
-
| ASN lookup | masked IP only (re-masked defensively in the resolver) |
|
|
126
|
-
| Anonymizer lookup | unmasked IP in, label out, nothing else retained |
|
|
127
|
-
| Private/localhost client (privacy-exempt) | no enrichment keys in env |
|
|
128
|
-
| Bad `*_db_path` | raises at boot, not per-request |
|
data/docs/geo-country.md
DELETED
|
@@ -1,181 +0,0 @@
|
|
|
1
|
-
# Geo-country resolution
|
|
2
|
-
|
|
3
|
-
Otto resolves a country-level ISO 3166-1 alpha-2 code for each request and
|
|
4
|
-
exposes it as `req.geo_country` / `env['otto.privacy.geo_country']`. Resolution
|
|
5
|
-
is country-only by design — that is the privacy posture; there is no city or
|
|
6
|
-
region lookup. (Two opt-in, database-only companion signals — ASN and
|
|
7
|
-
anonymizer classification — are covered in [enrichment](enrichment.md).)
|
|
8
|
-
|
|
9
|
-
## Resolution order
|
|
10
|
-
|
|
11
|
-
`Otto::Privacy::GeoResolver.resolve` returns the first hit from:
|
|
12
|
-
|
|
13
|
-
1. **Application-configured header** (`geo_header:`) — e.g. `X-Client-Country`.
|
|
14
|
-
2. **Known provider headers** — Cloudflare (`CF-IPCountry`), AWS CloudFront,
|
|
15
|
-
Fastly, Akamai Edgescape, Azure Front Door, **Vercel**
|
|
16
|
-
(`X-Vercel-IP-Country`), and a few semi-standard names
|
|
17
|
-
(`X-Geo-Country`, `X-Country-Code`, `Country-Code`).
|
|
18
|
-
3. **Custom resolver** (`GeoResolver.custom_resolver`) — your own callable.
|
|
19
|
-
Unlike the other geo settings, this is **class-level** (see
|
|
20
|
-
[Configuration](#configuration)).
|
|
21
|
-
4. **Local MMDB database** (`geo_db_path:` / `geo_db_reader:`) — a MaxMind-DB
|
|
22
|
-
country database.
|
|
23
|
-
5. **`'**'`** — the unknown sentinel, when nothing else matches.
|
|
24
|
-
|
|
25
|
-
Steps 1 and 2 are **only consulted when geo headers can be trusted** (see
|
|
26
|
-
[Header trust](#header-trust-and-spoofing) below).
|
|
27
|
-
|
|
28
|
-
Resolution is **honest**: Otto does not guess from a hardcoded IP-range table.
|
|
29
|
-
When no header, custom resolver, or database resolves a country, the result is
|
|
30
|
-
`'**'`.
|
|
31
|
-
|
|
32
|
-
### Privacy: masked IP and masked env
|
|
33
|
-
|
|
34
|
-
The database lookup in step 4 runs on the request's **masked** IP
|
|
35
|
-
(e.g. `203.0.113.0`), never the real address. `check_geo_database` masks the IP
|
|
36
|
-
internally with the config's `octet_precision` before the lookup, so even a
|
|
37
|
-
direct `GeoResolver.resolve` caller passing a real IP does not expose it to the
|
|
38
|
-
database. Country-level MMDB networks are almost always ≥ /24, so the default
|
|
39
|
-
/24-masked value (`octet_precision: 1`) resolves to the same country.
|
|
40
|
-
|
|
41
|
-
In the middleware path Otto additionally hands `resolve` a **masked env view**:
|
|
42
|
-
`REMOTE_ADDR`, `X-Forwarded-For`, `X-Real-IP`, `X-Client-IP`, and the RFC 7239
|
|
43
|
-
`Forwarded` header are masked. So a `custom_resolver` cannot read the raw client
|
|
44
|
-
IP out of `env` either — use the `ip` argument (already masked), not `env`.
|
|
45
|
-
|
|
46
|
-
> **`octet_precision: 2`** masks two octets (a /16). That is coarser than most
|
|
47
|
-
> country networks, so it can reduce database hit rate for the small share of
|
|
48
|
-
> countries whose ranges are finer than /16 — those requests fall through to
|
|
49
|
-
> `'**'`. Header and custom-resolver sources are unaffected (they ignore the
|
|
50
|
-
> IP). Keep the default precision if you rely on the MMDB fallback.
|
|
51
|
-
|
|
52
|
-
## Configuration
|
|
53
|
-
|
|
54
|
-
All geo configuration is **boot-time only** (set once during single-threaded
|
|
55
|
-
initialization, before serving requests), matching `custom_resolver`'s
|
|
56
|
-
contract. `geo_header`, `geo_db_path`, and `geo_db_reader` are stored on the
|
|
57
|
-
instance's `Otto::Privacy::Config`, so separate Otto instances hold independent
|
|
58
|
-
geo configuration.
|
|
59
|
-
|
|
60
|
-
> **`custom_resolver` is the exception — it is class-level, not per-instance.**
|
|
61
|
-
> `GeoResolver.custom_resolver=` sets a singleton on the `GeoResolver` class, so
|
|
62
|
-
> it is **shared across every Otto instance in the process** (last write wins).
|
|
63
|
-
> If you run multiple Otto instances that need different resolver strategies,
|
|
64
|
-
> the custom resolver cannot distinguish them — branch inside a single resolver
|
|
65
|
-
> on `env`, or use per-instance `geo_db_reader` instead.
|
|
66
|
-
|
|
67
|
-
```ruby
|
|
68
|
-
otto.configure_ip_privacy(
|
|
69
|
-
geo: true, # default; false disables geo entirely
|
|
70
|
-
geo_header: 'X-Client-Country', # trusted app header (optional)
|
|
71
|
-
geo_db_path: 'data/geo-whois-asn-country.mmdb', # local MMDB fallback (optional)
|
|
72
|
-
# geo_db_reader: MaxMind::DB.new(path), # or bring your own reader (optional)
|
|
73
|
-
)
|
|
74
|
-
```
|
|
75
|
-
|
|
76
|
-
- **`geo: false`** short-circuits everything: no header reads, and any loaded
|
|
77
|
-
database is unloaded from memory (`req.geo_country` becomes `nil`).
|
|
78
|
-
- **`geo_header:`** accepts either the HTTP header name (`X-Client-Country`) or
|
|
79
|
-
the Rack CGI env key (`HTTP_X_CLIENT_COUNTRY`), in any case, and is
|
|
80
|
-
canonicalized to the env-key form. Pass `''` to clear.
|
|
81
|
-
- **`geo_db_path:`** is loaded once at boot in `MODE_MEMORY`. An unreadable
|
|
82
|
-
path, a corrupt/non-MMDB file, or a missing `maxmind-db` gem raises
|
|
83
|
-
`ArgumentError` **at configuration time**, not per-request. Pass `''` to
|
|
84
|
-
unload.
|
|
85
|
-
- **`geo_db_reader:`** injects any object responding to `#get(ip)` (a
|
|
86
|
-
preconfigured `MaxMind::DB` reader or a test double), keeping the reader and
|
|
87
|
-
data-source choice independent of Otto. It **overrides** `geo_db_path` when
|
|
88
|
-
both are given in the same call; supplying `geo_db_path` alone in a later call
|
|
89
|
-
clears a prior reader override.
|
|
90
|
-
|
|
91
|
-
Each keyword follows a `nil` = "leave unchanged" contract; pass `''` to a header
|
|
92
|
-
or path to clear it. Any geo-affecting change triggers the boot-time database
|
|
93
|
-
(re)load, so a bad `geo_db_path` fails at the `configure_ip_privacy` call.
|
|
94
|
-
|
|
95
|
-
## The database: gem and datafile
|
|
96
|
-
|
|
97
|
-
The reader and the data file are independent — the MMDB format is the interop
|
|
98
|
-
point.
|
|
99
|
-
|
|
100
|
-
### Reader gem (`maxmind-db`)
|
|
101
|
-
|
|
102
|
-
The [`maxmind-db`](https://rubygems.org/gems/maxmind-db) gem (official MaxMind
|
|
103
|
-
reader, Apache-2.0, pure Ruby, zero runtime deps) is an **optional**
|
|
104
|
-
dependency. Otto only `require`s it when a database is configured. Add it to
|
|
105
|
-
your app when you use the database fallback:
|
|
106
|
-
|
|
107
|
-
```ruby
|
|
108
|
-
# Gemfile
|
|
109
|
-
gem 'maxmind-db', '~> 1.4'
|
|
110
|
-
```
|
|
111
|
-
|
|
112
|
-
### Data file (`geo-whois-asn-country`)
|
|
113
|
-
|
|
114
|
-
The recommended data file is
|
|
115
|
-
[`geo-whois-asn-country`](https://github.com/sapics/ip-location-db) from
|
|
116
|
-
sapics/ip-location-db: **PDDL v1.0 (public domain, no attribution required)**,
|
|
117
|
-
rebuilt daily, shipped as MMDB. Otto vendors no database — country data goes
|
|
118
|
-
stale, and a public-domain file you refresh on your own schedule keeps
|
|
119
|
-
licensing and freshness in your control.
|
|
120
|
-
|
|
121
|
-
Download it (IPv4+IPv6) into a path of your choosing:
|
|
122
|
-
|
|
123
|
-
```bash
|
|
124
|
-
mkdir -p data
|
|
125
|
-
curl -fsSL -o data/geo-whois-asn-country.mmdb \
|
|
126
|
-
https://github.com/sapics/ip-location-db/releases/download/latest/geo-whois-asn-country.mmdb
|
|
127
|
-
```
|
|
128
|
-
|
|
129
|
-
Refresh it on your own schedule (e.g. a daily cron job running the same curl).
|
|
130
|
-
Any MMDB country database works — GeoLite2-Country, DB-IP Country Lite,
|
|
131
|
-
iplocate, etc. — since `GeoResolver` tolerates the record shapes country
|
|
132
|
-
databases actually use: nested `country.iso_code` (GeoLite2-Country style), a
|
|
133
|
-
flat `country_code` string, and a bare-string `country`.
|
|
134
|
-
|
|
135
|
-
> **Note on GeoLite2:** its EULA requires a MaxMind account/license key and
|
|
136
|
-
> obliges consumers to refresh within 30 days of each release. A PDDL dataset
|
|
137
|
-
> avoids both obligations.
|
|
138
|
-
|
|
139
|
-
## Header trust and spoofing
|
|
140
|
-
|
|
141
|
-
Every geo header is trivially client-spoofable unless the request actually
|
|
142
|
-
arrived through the CDN that sets it. Otto trusts geo headers — both the
|
|
143
|
-
configured `geo_header` and the provider headers — **only** for a request that
|
|
144
|
-
demonstrably arrived via a configured **CIDR trusted proxy**
|
|
145
|
-
(`env['otto.via_trusted_proxy']` with `trusted_proxies` configured). A spoofed
|
|
146
|
-
header on a direct connection is ignored, and resolution falls through to the
|
|
147
|
-
custom resolver / database.
|
|
148
|
-
|
|
149
|
-
Origins Otto cannot verify are **not** trusted:
|
|
150
|
-
|
|
151
|
-
- **No trusted-proxy configuration.** A direct internet client could otherwise
|
|
152
|
-
pick its own country by sending `CF-IPCountry` / `X-Client-Country`, so with
|
|
153
|
-
no `trusted_proxies` configured, header steps are skipped and resolution falls
|
|
154
|
-
to the resolver / database (`'**'` if neither is set).
|
|
155
|
-
- **Count-based `trusted_proxy_depth` mode.** The header-setting hop cannot be
|
|
156
|
-
verified as a geo-CDN, so depth mode does not enable header trust. This
|
|
157
|
-
conflict fails loud: configuring a `geo_header` together with a
|
|
158
|
-
`trusted_proxy_depth` raises `ArgumentError` at configuration time (in
|
|
159
|
-
either order) instead of silently ignoring the header per-request.
|
|
160
|
-
Database-backed geo remains fully supported under depth. The built-in
|
|
161
|
-
provider headers stay legal (there is nothing to configure, so nothing
|
|
162
|
-
can raise) but are equally inert — header trust requires CIDR-verified
|
|
163
|
-
proxies — so only an explicitly configured `geo_header` is rejected, and
|
|
164
|
-
depth deployments that want geo should set `geo_db_path`.
|
|
165
|
-
|
|
166
|
-
**Migration:** to keep header-based geo, configure `trusted_proxies` (CIDR
|
|
167
|
-
matchers) so Otto can verify the proxy origin. Depth-mode and header-only
|
|
168
|
-
deployments should set `geo_db_path` for a local database instead; otherwise
|
|
169
|
-
resolution returns `'**'`.
|
|
170
|
-
|
|
171
|
-
## Acceptance behavior summary
|
|
172
|
-
|
|
173
|
-
| Scenario | Result |
|
|
174
|
-
| --- | --- |
|
|
175
|
-
| Configured `geo_header` present and trusted | wins over provider headers |
|
|
176
|
-
| Request not via a verified CIDR trusted proxy | geo headers skipped |
|
|
177
|
-
| No `trusted_proxies` configured | geo headers skipped (not trusted) |
|
|
178
|
-
| Database lookup | uses the masked IP only |
|
|
179
|
-
| `geo: false` | `nil`, no database in memory |
|
|
180
|
-
| Bad `geo_db_path` | raises at boot, not per-request |
|
|
181
|
-
| Nothing matches | `'**'` |
|