otto 2.9.0 → 2.10.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (96) hide show
  1. checksums.yaml +4 -4
  2. data/.github/dependabot.yml +5 -0
  3. data/.github/workflows/ci.yml +11 -8
  4. data/.github/workflows/claude-code-review.yml +38 -13
  5. data/.github/workflows/claude.yml +10 -8
  6. data/.github/workflows/code-smells.yml +5 -5
  7. data/.github/workflows/release-gem.yml +2 -2
  8. data/.github/workflows/ruby-lint.yml +3 -3
  9. data/.github/workflows/yardoc.yml +5 -5
  10. data/.gitignore +1 -5
  11. data/.rubocop_todo.yml +7 -5
  12. data/AGENTS.md +22 -1
  13. data/CHANGELOG.rst +203 -0
  14. data/Gemfile +3 -3
  15. data/Gemfile.lock +11 -15
  16. data/README.md +75 -37
  17. data/docs/README.md +166 -0
  18. data/docs/adr/README.md +16 -0
  19. data/docs/adr/adr-001-route-authentication-at-handler-boundary.md +38 -0
  20. data/docs/adr/adr-002-multi-strategy-authentication-and-authorization.md +50 -0
  21. data/docs/adr/adr-003-caddy-tls-route-based-integration.md +48 -0
  22. data/docs/adr/adr-004-separate-compatibility-from-security-maintenance.md +58 -0
  23. data/docs/guides/authentication.md +377 -0
  24. data/docs/guides/caddy-tls.md +205 -0
  25. data/docs/guides/configuration_freezing.md +146 -0
  26. data/docs/guides/enrichment.md +161 -0
  27. data/docs/guides/forwarded-authority.md +249 -0
  28. data/docs/guides/geo-country.md +168 -0
  29. data/docs/guides/ip_privacy.md +39 -0
  30. data/docs/guides/ipaddr-encoding-quirk.md +56 -0
  31. data/docs/guides/mcp.md +282 -0
  32. data/docs/guides/privacy.md +193 -0
  33. data/docs/guides/routing.md +181 -0
  34. data/docs/guides/structured_logging.md +281 -0
  35. data/docs/guides/testing-guide.md +391 -0
  36. data/docs/maintainers/github-actions.md +41 -0
  37. data/docs/maintainers/investigations/.gitignore +2 -0
  38. data/docs/migrating/v2.0.0.md +337 -0
  39. data/docs/reference/authentication.md +290 -0
  40. data/docs/reference/route-syntax.md +181 -0
  41. data/docs/reference/runtime-and-dependency-security.md +86 -0
  42. data/examples/advanced_routes/README.md +43 -57
  43. data/examples/authentication_strategies/README.md +37 -196
  44. data/examples/basic/README.md +24 -39
  45. data/examples/basic/config.ru +0 -1
  46. data/examples/caddy_tls_demo/README.md +8 -2
  47. data/examples/lambda_handlers/README.md +11 -2
  48. data/examples/mcp_demo/README.md +75 -161
  49. data/examples/mcp_demo/config.ru +1 -0
  50. data/examples/security_features/README.md +36 -234
  51. data/lib/otto/caddy_tls/localhost_guard.rb +22 -24
  52. data/lib/otto/core/configuration.rb +36 -22
  53. data/lib/otto/core/file_safety.rb +89 -31
  54. data/lib/otto/core/middleware_stack.rb +36 -36
  55. data/lib/otto/core/router.rb +62 -19
  56. data/lib/otto/env_keys.rb +20 -2
  57. data/lib/otto/mcp/auth/token.rb +10 -4
  58. data/lib/otto/mcp/core.rb +23 -5
  59. data/lib/otto/mcp/endpoint.rb +41 -0
  60. data/lib/otto/mcp/errors.rb +15 -0
  61. data/lib/otto/mcp/options.rb +292 -0
  62. data/lib/otto/mcp/protocol.rb +52 -22
  63. data/lib/otto/mcp/rate_limiting.rb +175 -97
  64. data/lib/otto/mcp/registry.rb +14 -14
  65. data/lib/otto/mcp/schema_validation.rb +20 -12
  66. data/lib/otto/mcp/server.rb +131 -32
  67. data/lib/otto/optional_dependency.rb +57 -0
  68. data/lib/otto/privacy/config.rb +10 -8
  69. data/lib/otto/security/authentication/route_auth_wrapper/role_authorization.rb +22 -5
  70. data/lib/otto/security/authentication/strategies/api_key_strategy.rb +181 -18
  71. data/lib/otto/security/authentication/strategies/permission_strategy.rb +1 -0
  72. data/lib/otto/security/authentication/strategies/role_strategy.rb +1 -0
  73. data/lib/otto/security/authentication/strategies/session_strategy.rb +1 -0
  74. data/lib/otto/security/authentication/strategy_result.rb +58 -28
  75. data/lib/otto/security/config.rb +328 -14
  76. data/lib/otto/security/configurator.rb +36 -6
  77. data/lib/otto/security/core.rb +19 -1
  78. data/lib/otto/security/middleware/ip_privacy_middleware.rb +105 -4
  79. data/lib/otto/security/middleware/rate_limit_middleware.rb +1 -6
  80. data/lib/otto/security/rate_limiter.rb +81 -46
  81. data/lib/otto/utils.rb +50 -0
  82. data/lib/otto/version.rb +1 -1
  83. data/lib/otto.rb +9 -11
  84. data/otto.gemspec +0 -2
  85. metadata +31 -41
  86. data/docs/.gitignore +0 -10
  87. data/docs/1108-STREAMING_ARCHITECTURE_ANALYSIS.md +0 -1105
  88. data/docs/1108-STREAMING_SUPPORT_SUMMARY.md +0 -376
  89. data/docs/enrichment.md +0 -128
  90. data/docs/geo-country.md +0 -181
  91. data/docs/ipaddr-encoding-quirk.md +0 -34
  92. data/docs/migrating/v2.0.0-pre1.md +0 -276
  93. data/docs/migrating/v2.0.0-pre2.md +0 -338
  94. data/docs/modern-authentication-authorization-landscape.md +0 -558
  95. data/docs/multi-strategy-authentication-design.md +0 -1401
  96. data/docs/reverse-proxy-network-services.md +0 -371
@@ -1,376 +0,0 @@
1
- # Otto Streaming Support: Executive Summary
2
-
3
- **Investigation Date**: 2025-11-08
4
- **Question**: Should Otto support Server-Sent Events (SSE) and WebSockets?
5
- **Answer**: **No** - Use separate services or long-polling instead
6
-
7
- > **Note**: The Ruby snippets below illustrate the recommended architecture. They
8
- > are written against the current Otto Logic-class contract
9
- > (`initialize(context, params, locale)` + `process`), but have not been executed
10
- > end-to-end. Runnable examples are tracked separately.
11
-
12
- ---
13
-
14
- ## Quick Recommendation
15
-
16
- | Use Case | Solution | Complexity | Otto Compatible? |
17
- |----------|----------|------------|------------------|
18
- | **Low-frequency updates (<1/min)** | Long-polling | ⭐ Simple | ✅ Yes |
19
- | **Medium-frequency updates (1-10/sec)** | Separate SSE service | ⭐⭐ Moderate | ✅ Via integration |
20
- | **High-frequency updates (>10/sec)** | Separate WebSocket service | ⭐⭐⭐ Complex | ✅ Via integration |
21
- | **Bidirectional communication** | Separate WebSocket service | ⭐⭐⭐ Complex | ✅ Via integration |
22
-
23
- ---
24
-
25
- ## Key Findings
26
-
27
- ### 1. **Otto's Architecture is Fundamentally Incompatible with Streaming**
28
-
29
- Otto is designed as a **stateless, synchronous, request/response** framework:
30
-
31
- ```
32
- Request → Middleware → Route → Handler → Response → Close Connection
33
- ```
34
-
35
- SSE/WebSocket require **stateful, long-lived connections**:
36
-
37
- ```
38
- Request → Upgrade → Keep Open → Stream Data (minutes/hours) → Close
39
- ```
40
-
41
- **Incompatibilities**:
42
- - ❌ Response handlers expect complete responses (not streaming enumerators)
43
- - ❌ Middleware stack can't unwind during long-lived connections
44
- - ❌ Configuration freezing prevents runtime streaming adjustments
45
- - ❌ Requires async servers (Falcon, Iodine) - breaks server-agnostic design
46
- - ❌ Stateful routing complicates horizontal scaling
47
-
48
- ### 2. **Industry Best Practice: Separate Services**
49
-
50
- Modern frameworks separate real-time communication from REST APIs:
51
-
52
- **Rails (ActionCable)**:
53
- ```
54
- Rails App (Puma) → Redis Pub/Sub ← ActionCable (Separate Process)
55
- ```
56
-
57
- **Node.js (Express + Socket.IO)**:
58
- ```
59
- Express (HTTP Routes) + Socket.IO (Separate Layer)
60
- ```
61
-
62
- **Benefits**:
63
- - ✅ Independent scaling (scale WebSocket separately from API)
64
- - ✅ Technology choice (use best tool for each job)
65
- - ✅ Fault isolation (WebSocket crash doesn't affect API)
66
- - ✅ Clear architectural boundaries
67
-
68
- ### 3. **Long-Polling Works Perfectly with Otto**
69
-
70
- For low-to-medium frequency updates, long-polling is **simple and effective**:
71
-
72
- ```ruby
73
- # Otto route (works with any Rack server)
74
- GET /api/notifications/poll NotificationPollLogic response=json auth=session
75
-
76
- class NotificationPollLogic
77
- attr_reader :context, :params, :locale
78
-
79
- def initialize(context, params, locale)
80
- @context = context
81
- @params = params
82
- @locale = locale
83
- end
84
-
85
- def process
86
- # Logic-class params are string-keyed (LogicClassHandler does not apply
87
- # Otto::Static.indifferent_params).
88
- timeout = params['timeout'].to_i.clamp(1, 30)
89
- last_id = params['last_id'].to_i
90
- start_time = Time.now
91
-
92
- # Identity comes from the StrategyResult, never from request params.
93
- user_id = context.user_id
94
-
95
- loop do
96
- notifications = fetch_new(user_id, last_id)
97
- return { notifications: notifications } if notifications.any?
98
-
99
- break if Time.now - start_time > timeout
100
-
101
- sleep 0.5
102
- end
103
-
104
- { notifications: [] }
105
- end
106
- end
107
- ```
108
-
109
- **Benefits**:
110
- - ✅ HTTP-based (cacheable, proxy-friendly, standard tooling)
111
- - ✅ Works with Otto's synchronous model
112
- - ✅ Compatible with any Rack server (Puma, Unicorn, Passenger)
113
- - ✅ Simple debugging (standard HTTP requests/responses)
114
- - ✅ No external dependencies
115
-
116
- ---
117
-
118
- ## Recommended Solutions
119
-
120
- ### Option 1: Long-Polling (SIMPLEST)
121
-
122
- **When to use**:
123
- - Updates less than 1 per minute
124
- - Moderate concurrency (<10,000 clients)
125
- - Simple deployment preferred
126
-
127
- **Example**: see the `NotificationPollLogic` sketch above.
128
-
129
- **Complexity**: ⭐ Simple
130
- **Otto Integration**: ✅ Native support (no changes needed)
131
-
132
- ---
133
-
134
- ### Option 2: Separate SSE Service (RECOMMENDED FOR REAL-TIME)
135
-
136
- **When to use**:
137
- - Updates 1-10 per second
138
- - High concurrency (>10,000 clients)
139
- - Near-instant updates required (<100ms latency)
140
-
141
- **Architecture**:
142
- ```
143
- ┌─────────────────┐
144
- │ Otto API │ ← Stateless HTTP (authentication, business logic)
145
- │ (Puma) │
146
- └─────────────────┘
147
-
148
- ┌─────────┐
149
- │ Redis │ ← Message queue (pub/sub)
150
- │ Pub/Sub │
151
- └─────────┘
152
-
153
- ┌─────────────────┐
154
- │ SSE Service │ ← Stateful streaming (Falcon/Iodine)
155
- │ (Falcon) │
156
- └─────────────────┘
157
- ```
158
-
159
- **Example**: see "Otto + Falcon SSE Integration" in
160
- `docs/1108-STREAMING_ARCHITECTURE_ANALYSIS.md`.
161
-
162
- **Complexity**: ⭐⭐ Moderate
163
- **Otto Integration**: ✅ Via Redis pub/sub
164
-
165
- ---
166
-
167
- ### Option 3: Third-Party Service (COMMERCIAL)
168
-
169
- **When to use**:
170
- - Don't want to manage WebSocket infrastructure
171
- - Need global CDN distribution
172
- - Require guaranteed SLA
173
-
174
- **Options**:
175
- - **Mercure**: Open-source SSE hub (self-hosted or managed)
176
- - **Ably**: Commercial real-time messaging platform
177
- - **Pusher**: Commercial WebSocket/SSE service
178
-
179
- **Complexity**: ⭐⭐ Moderate (integration)
180
- **Otto Integration**: ✅ Via HTTP API
181
-
182
- ---
183
-
184
- ## Documentation Created
185
-
186
- ### **docs/1108-STREAMING_ARCHITECTURE_ANALYSIS.md** (15,000+ words)
187
- Comprehensive technical analysis covering:
188
- - Otto's current architecture (detailed lifecycle analysis)
189
- - Technical requirements for SSE/WebSocket (Rack 3, hijacking, etc.)
190
- - Industry patterns (Rails, Sinatra, Roda, Go, Node.js)
191
- - Compatibility analysis (why it doesn't fit)
192
- - Best practices and anti-patterns
193
- - Detailed recommendations with code examples
194
-
195
- ### Runnable examples: not yet published
196
-
197
- Draft `examples/otto_falcon_sse_integration.rb` and
198
- `examples/long_polling_example.rb` were written alongside this analysis but
199
- targeted an Otto API surface that does not exist (`Otto::RequestContext`,
200
- `enable_sessions!`, `raise_concern`, a `session` helper, `Rack::Handler::Puma`
201
- under the Rack 3 pin) and used symbol-keyed params that Logic routes never
202
- populate. They were withheld rather than shipped as copy-paste material; the
203
- architecture they demonstrated is preserved in the snippets here and in the
204
- analysis document.
205
-
206
- ---
207
-
208
- ## Key Insights
209
-
210
- ### The Real Question
211
-
212
- **Not**: "Can Otto support SSE/WebSocket?"
213
- (Technically possible with massive refactoring)
214
-
215
- **But**: "Should Otto support SSE/WebSocket?"
216
- (Architecturally inadvisable)
217
-
218
- ### Answer: **No**
219
-
220
- Otto should remain focused on its core strengths:
221
- - ✅ **Stateless** HTTP APIs
222
- - ✅ **Security-first** design (CSRF, rate limiting, validation)
223
- - ✅ **Privacy by default** (IP masking, geo-location)
224
- - ✅ **Server-agnostic** (works with any Rack server)
225
- - ✅ **Simple** and predictable architecture
226
-
227
- Adding SSE/WebSocket would:
228
- - ❌ Compromise architectural integrity
229
- - ❌ Force specific async servers (Falcon, Iodine)
230
- - ❌ Complicate security guarantees (middleware assumptions broken)
231
- - ❌ Add significant complexity for niche use case
232
- - ❌ Go against industry best practices (separation of concerns)
233
-
234
- ---
235
-
236
- ## What Otto SHOULD Do
237
-
238
- ### 1. ✅ Document Integration Patterns
239
-
240
- Add official guide: "Integrating Otto with Real-Time Services"
241
- - Long-polling patterns (built-in support)
242
- - Separate SSE service pattern (Otto + Falcon + Redis)
243
- - Third-party service integration (Mercure, Ably, Pusher)
244
-
245
- ### 2. ✅ Provide Example Code
246
-
247
- Add to `examples/` directory, written against the real Logic-class contract and
248
- verified to boot:
249
- - `long_polling_example.rb`
250
- - `otto_falcon_sse_integration.rb`
251
- - `otto_mercure_integration.rb`
252
-
253
- ### 3. ⚠️ Consider Plugin System (If Community Demands)
254
-
255
- **Only if there's strong demand**, create experimental plugin:
256
- - Clearly marked "experimental" and "unsupported"
257
- - Requires Falcon/Iodine (documented)
258
- - Security implications documented
259
- - No core changes required
260
-
261
- ### 4. ❌ Do NOT Add to Core
262
-
263
- Preserve Otto's architectural integrity by:
264
- - Keeping core stateless and synchronous
265
- - Maintaining server-agnostic design
266
- - Focusing on security and simplicity
267
- - Following industry best practices (separation of concerns)
268
-
269
- ---
270
-
271
- ## Migration Guide for Existing Users
272
-
273
- If you currently need real-time updates:
274
-
275
- ### Step 1: Assess Your Use Case
276
-
277
- **Low-frequency updates (<1/min)**:
278
- → Use long-polling (Otto native support)
279
-
280
- **Medium-frequency updates (1-10/sec)**:
281
- → Use separate SSE service (Otto + Falcon + Redis)
282
-
283
- **High-frequency or bidirectional**:
284
- → Use separate WebSocket service or commercial solution
285
-
286
- ### Step 2: Implementation Path
287
-
288
- #### For Long-Polling:
289
- 1. Create Otto route with long-polling logic
290
- 2. Use `sleep` loop with timeout
291
- 3. Client polls with timeout parameter
292
- 4. No external dependencies needed
293
-
294
- See the `NotificationPollLogic` sketch above.
295
-
296
- #### For Separate SSE Service:
297
- 1. Otto API handles authentication and publishes to Redis
298
- 2. Separate Falcon app subscribes to Redis and streams SSE
299
- 3. Client connects to SSE service with JWT token from Otto
300
- 4. Scale Otto and SSE services independently
301
-
302
- See "Otto + Falcon SSE Integration" in
303
- `docs/1108-STREAMING_ARCHITECTURE_ANALYSIS.md`.
304
-
305
- ### Step 3: Deployment
306
-
307
- **Long-Polling**:
308
- - Deploy with existing Otto setup (Puma, Unicorn, Passenger)
309
- - Increase thread pool size for long-polling routes
310
- - Monitor connection pool (ensure enough threads)
311
-
312
- **Separate SSE Service**:
313
- - Deploy Otto API with Puma (standard)
314
- - Deploy Falcon SSE service separately (dedicated servers)
315
- - Use Redis for pub/sub (cluster-ready)
316
- - Configure Nginx with sticky sessions for SSE
317
- - Scale services independently based on load
318
-
319
- ---
320
-
321
- ## Performance Guidance
322
-
323
- ### Long-Polling Capacity
324
-
325
- **Example**: Puma with 5 workers × 32 threads = 160 concurrent requests
326
-
327
- If long-polling uses 30s timeout:
328
- - 160 concurrent connections
329
- - ~320 clients with 50% utilization
330
- - Up to 10,000 clients with proper thread tuning
331
-
332
- **Good for**: Dashboard metrics, low-volume notifications
333
-
334
- ### SSE/WebSocket Capacity
335
-
336
- **Example**: Falcon with 4 workers (async)
337
-
338
- Each worker handles thousands of concurrent connections via fibers:
339
- - 10,000+ concurrent SSE connections per server
340
- - Horizontal scaling via Redis pub/sub
341
- - Near-instant message delivery
342
-
343
- **Good for**: Chat, multiplayer, high-frequency updates
344
-
345
- ---
346
-
347
- ## Conclusion
348
-
349
- **Otto should NOT integrate SSE/WebSocket support** because:
350
-
351
- 1. **Architectural mismatch**: Stateless vs stateful paradigms
352
- 2. **Industry consensus**: Separate services is best practice
353
- 3. **Complexity cost**: Massive refactoring for niche use case
354
- 4. **Better alternatives**: Long-polling (simple) or separate services (powerful)
355
-
356
- **Instead, Otto should**:
357
-
358
- 1. ✅ Document long-polling patterns (works today)
359
- 2. ✅ Provide integration examples (Otto + Falcon + Redis)
360
- 3. ✅ Recommend third-party solutions (Mercure, Ably, Pusher)
361
- 4. ✅ Stay focused on stateless HTTP APIs
362
-
363
- **This preserves Otto's core strengths** while enabling users who need real-time functionality to integrate appropriate solutions.
364
-
365
- ---
366
-
367
- ## Further Reading
368
-
369
- - **Technical Analysis**: `docs/1108-STREAMING_ARCHITECTURE_ANALYSIS.md`
370
- - **Rack 3 Streaming**: https://github.com/rack/rack/issues/1600
371
- - **ActionCable Architecture**: https://guides.rubyonrails.org/action_cable_overview.html
372
- - **SSE vs WebSocket**: https://ably.com/blog/websockets-vs-sse
373
-
374
- ---
375
-
376
- **End of Summary**
data/docs/enrichment.md DELETED
@@ -1,128 +0,0 @@
1
- # ASN and anonymizer enrichment
2
-
3
- Two opt-in, country-adjacent signals layered on the same privacy pipeline as
4
- [geo-country](geo-country.md):
5
-
6
- - **ASN** — the network operator an address belongs to, as `req.asn` /
7
- `env['otto.privacy.asn']`: `'AS15169'`.
8
- - **Anonymizer** — whether the address is a known anonymizing egress, as
9
- `req.anonymizer` / `env['otto.privacy.anonymizer']`: `'tor'`, `'proxy'`,
10
- `'vpn'`, `'residential_proxy'`, `'hosting'`, `'anonymous'`, `'none'`, `'**'`.
11
-
12
- Both are **off by default** (unlike geo) and **database-only**: no CDN
13
- publishes a client-ASN or anonymizer header with meaningful deployment, so
14
- there is no header tier and none of geo's header-trust machinery applies —
15
- including the `geo_header`/`trusted_proxy_depth` boot conflict.
16
-
17
- Every signal keeps the same three-state contract:
18
-
19
- | Value | Meaning |
20
- | --- | --- |
21
- | `nil` | the signal is switched off |
22
- | `'**'` | switched on, but nothing resolved (no database, or lookup failed) |
23
- | a label | a real answer |
24
-
25
- ## Configuration
26
-
27
- ```ruby
28
- otto.configure_ip_privacy(
29
- asn: true,
30
- asn_db_path: 'data/origin-asn.mmdb',
31
- anonymizer: true,
32
- anonymizer_db_path: 'data/anonymizer.mmdb',
33
- )
34
- ```
35
-
36
- Both accept the same bring-your-own-reader seam as geo (`asn_db_reader:` /
37
- `anonymizer_db_reader:` — any object responding to `#get(ip)`; a reader
38
- supplied in the same call wins over a path). Bad paths raise at boot, not
39
- per-request. The [`maxmind-db`](https://rubygems.org/gems/maxmind-db) gem is
40
- required only when a `*_db_path` is configured.
41
-
42
- ## ASN: which address is looked up, and why that's safe
43
-
44
- The ASN lookup uses the **masked** IP, same as geo. This is not a compromise:
45
- IPv4 BGP routes are not announced longer than /24, so a /24-masked address
46
- falls inside the same announced prefix — and therefore the same ASN — as the
47
- real one. (IPv6 is coarser: at `octet_precision: 1` Otto zeroes the last 80
48
- bits, wider than many IPv6 announcements, so treat IPv6 ASN as best-effort.)
49
-
50
- ### Data file
51
-
52
- > **Naming caution:** the country database this project recommends,
53
- > `geo-whois-asn-country`, does **not** contain ASN data. In
54
- > sapics/ip-location-db naming, `geo-whois-asn` describes the *sources* the
55
- > country data was derived from; the final token (`-country`) is what the
56
- > records contain. ASN data is a separate file type in that project.
57
-
58
- The recommended ASN file is
59
- [`origin-asn`](https://github.com/sapics/ip-location-db/tree/main/origin-asn/)
60
- from sapics/ip-location-db — like `geo-whois-asn-country` it is **PDDL v1.0
61
- (public domain)** and rebuilt daily, so the licensing/freshness posture
62
- matches the geo guidance:
63
-
64
- ```bash
65
- curl -fsSL -o data/origin-asn.mmdb \
66
- https://github.com/sapics/ip-location-db/releases/download/latest/origin-asn.mmdb
67
- ```
68
-
69
- Its records carry a flat `autonomous_system_number` (verified against the
70
- published file), which is the primary key `AsnResolver` reads. GeoLite2-ASN
71
- and DB-IP ASN Lite MMDBs work too (same key; GeoLite2's EULA caveats from the
72
- geo doc apply). Reserved ASNs (0 per RFC 7607, the AS_TRANS placeholder 23456
73
- per RFC 6793) resolve to `'**'` rather than being reported as operators.
74
-
75
- ## Anonymizer: the one unmasked lookup
76
-
77
- Anonymizer classification reads the **unmasked** address. This is deliberate
78
- and documented in `AnonymizerResolver` itself: anonymizer databases list
79
- individual egress nodes at or near /32, so the /24 equivalence that justifies
80
- masked geo and ASN lookups does not hold. A masked lookup would flag a whole
81
- /24 because one host in it runs a Tor exit, and miss the exit node itself —
82
- wrong in both directions.
83
-
84
- The privacy containment is the same one Otto already relies on for
85
- `hash_ip` and `env['otto.ip_match']`, which also consume the full IP: **only
86
- the derived value leaves**. The resolver returns a label; the address is
87
- never persisted, serialized, or handed downstream.
88
-
89
- ### Reading the labels
90
-
91
- When several database flags are set at once (a Tor exit hosted at a cloud
92
- provider), the **most specific** label wins: `tor` > `proxy` > `vpn` >
93
- `residential_proxy` > `hosting` > `anonymous`.
94
-
95
- Two labels deserve care:
96
-
97
- - **`'none'`** means the database was consulted and does not list the
98
- address. For an anonymizer database that is a real answer (these files
99
- record only flagged addresses), but it is *not* a positive assertion the
100
- visitor is residential — it is only as fresh as your database file.
101
- - **`'**'`** means no database answered at all. Do not collapse it into
102
- `'none'`: `'none'` is evidence, `'**'` is the absence of evidence. A
103
- "block anonymizers" rule that treats `'**'` as `'none'` fails open when
104
- the database file goes missing.
105
-
106
- ### Data file
107
-
108
- There is no public-domain anonymizer dataset of `origin-asn`'s quality; this
109
- signal is bring-your-own-database. `AnonymizerResolver` reads the MaxMind
110
- GeoIP2 Anonymous-IP flag schema (`is_tor_exit_node`, `is_public_proxy`,
111
- `is_anonymous_vpn`, `is_residential_proxy`, `is_hosting_provider`,
112
- `is_anonymous`), which commercial and self-built MMDBs alike use. A workable
113
- self-built option: compile the [Tor bulk exit
114
- list](https://check.torproject.org/torbulkexitlist) into an MMDB with
115
- `is_tor_exit_node` set — that covers the highest-signal label with fully
116
- public data.
117
-
118
- ## Acceptance behavior summary
119
-
120
- | Scenario | Result |
121
- | --- | --- |
122
- | Signal not enabled | `nil` everywhere (env key absent for exempt IPs) |
123
- | Enabled, no database configured | `'**'` |
124
- | Database read raises | `'**'` (a lookup must never crash a request) |
125
- | ASN lookup | masked IP only (re-masked defensively in the resolver) |
126
- | Anonymizer lookup | unmasked IP in, label out, nothing else retained |
127
- | Private/localhost client (privacy-exempt) | no enrichment keys in env |
128
- | Bad `*_db_path` | raises at boot, not per-request |
data/docs/geo-country.md DELETED
@@ -1,181 +0,0 @@
1
- # Geo-country resolution
2
-
3
- Otto resolves a country-level ISO 3166-1 alpha-2 code for each request and
4
- exposes it as `req.geo_country` / `env['otto.privacy.geo_country']`. Resolution
5
- is country-only by design — that is the privacy posture; there is no city or
6
- region lookup. (Two opt-in, database-only companion signals — ASN and
7
- anonymizer classification — are covered in [enrichment](enrichment.md).)
8
-
9
- ## Resolution order
10
-
11
- `Otto::Privacy::GeoResolver.resolve` returns the first hit from:
12
-
13
- 1. **Application-configured header** (`geo_header:`) — e.g. `X-Client-Country`.
14
- 2. **Known provider headers** — Cloudflare (`CF-IPCountry`), AWS CloudFront,
15
- Fastly, Akamai Edgescape, Azure Front Door, **Vercel**
16
- (`X-Vercel-IP-Country`), and a few semi-standard names
17
- (`X-Geo-Country`, `X-Country-Code`, `Country-Code`).
18
- 3. **Custom resolver** (`GeoResolver.custom_resolver`) — your own callable.
19
- Unlike the other geo settings, this is **class-level** (see
20
- [Configuration](#configuration)).
21
- 4. **Local MMDB database** (`geo_db_path:` / `geo_db_reader:`) — a MaxMind-DB
22
- country database.
23
- 5. **`'**'`** — the unknown sentinel, when nothing else matches.
24
-
25
- Steps 1 and 2 are **only consulted when geo headers can be trusted** (see
26
- [Header trust](#header-trust-and-spoofing) below).
27
-
28
- Resolution is **honest**: Otto does not guess from a hardcoded IP-range table.
29
- When no header, custom resolver, or database resolves a country, the result is
30
- `'**'`.
31
-
32
- ### Privacy: masked IP and masked env
33
-
34
- The database lookup in step 4 runs on the request's **masked** IP
35
- (e.g. `203.0.113.0`), never the real address. `check_geo_database` masks the IP
36
- internally with the config's `octet_precision` before the lookup, so even a
37
- direct `GeoResolver.resolve` caller passing a real IP does not expose it to the
38
- database. Country-level MMDB networks are almost always ≥ /24, so the default
39
- /24-masked value (`octet_precision: 1`) resolves to the same country.
40
-
41
- In the middleware path Otto additionally hands `resolve` a **masked env view**:
42
- `REMOTE_ADDR`, `X-Forwarded-For`, `X-Real-IP`, `X-Client-IP`, and the RFC 7239
43
- `Forwarded` header are masked. So a `custom_resolver` cannot read the raw client
44
- IP out of `env` either — use the `ip` argument (already masked), not `env`.
45
-
46
- > **`octet_precision: 2`** masks two octets (a /16). That is coarser than most
47
- > country networks, so it can reduce database hit rate for the small share of
48
- > countries whose ranges are finer than /16 — those requests fall through to
49
- > `'**'`. Header and custom-resolver sources are unaffected (they ignore the
50
- > IP). Keep the default precision if you rely on the MMDB fallback.
51
-
52
- ## Configuration
53
-
54
- All geo configuration is **boot-time only** (set once during single-threaded
55
- initialization, before serving requests), matching `custom_resolver`'s
56
- contract. `geo_header`, `geo_db_path`, and `geo_db_reader` are stored on the
57
- instance's `Otto::Privacy::Config`, so separate Otto instances hold independent
58
- geo configuration.
59
-
60
- > **`custom_resolver` is the exception — it is class-level, not per-instance.**
61
- > `GeoResolver.custom_resolver=` sets a singleton on the `GeoResolver` class, so
62
- > it is **shared across every Otto instance in the process** (last write wins).
63
- > If you run multiple Otto instances that need different resolver strategies,
64
- > the custom resolver cannot distinguish them — branch inside a single resolver
65
- > on `env`, or use per-instance `geo_db_reader` instead.
66
-
67
- ```ruby
68
- otto.configure_ip_privacy(
69
- geo: true, # default; false disables geo entirely
70
- geo_header: 'X-Client-Country', # trusted app header (optional)
71
- geo_db_path: 'data/geo-whois-asn-country.mmdb', # local MMDB fallback (optional)
72
- # geo_db_reader: MaxMind::DB.new(path), # or bring your own reader (optional)
73
- )
74
- ```
75
-
76
- - **`geo: false`** short-circuits everything: no header reads, and any loaded
77
- database is unloaded from memory (`req.geo_country` becomes `nil`).
78
- - **`geo_header:`** accepts either the HTTP header name (`X-Client-Country`) or
79
- the Rack CGI env key (`HTTP_X_CLIENT_COUNTRY`), in any case, and is
80
- canonicalized to the env-key form. Pass `''` to clear.
81
- - **`geo_db_path:`** is loaded once at boot in `MODE_MEMORY`. An unreadable
82
- path, a corrupt/non-MMDB file, or a missing `maxmind-db` gem raises
83
- `ArgumentError` **at configuration time**, not per-request. Pass `''` to
84
- unload.
85
- - **`geo_db_reader:`** injects any object responding to `#get(ip)` (a
86
- preconfigured `MaxMind::DB` reader or a test double), keeping the reader and
87
- data-source choice independent of Otto. It **overrides** `geo_db_path` when
88
- both are given in the same call; supplying `geo_db_path` alone in a later call
89
- clears a prior reader override.
90
-
91
- Each keyword follows a `nil` = "leave unchanged" contract; pass `''` to a header
92
- or path to clear it. Any geo-affecting change triggers the boot-time database
93
- (re)load, so a bad `geo_db_path` fails at the `configure_ip_privacy` call.
94
-
95
- ## The database: gem and datafile
96
-
97
- The reader and the data file are independent — the MMDB format is the interop
98
- point.
99
-
100
- ### Reader gem (`maxmind-db`)
101
-
102
- The [`maxmind-db`](https://rubygems.org/gems/maxmind-db) gem (official MaxMind
103
- reader, Apache-2.0, pure Ruby, zero runtime deps) is an **optional**
104
- dependency. Otto only `require`s it when a database is configured. Add it to
105
- your app when you use the database fallback:
106
-
107
- ```ruby
108
- # Gemfile
109
- gem 'maxmind-db', '~> 1.4'
110
- ```
111
-
112
- ### Data file (`geo-whois-asn-country`)
113
-
114
- The recommended data file is
115
- [`geo-whois-asn-country`](https://github.com/sapics/ip-location-db) from
116
- sapics/ip-location-db: **PDDL v1.0 (public domain, no attribution required)**,
117
- rebuilt daily, shipped as MMDB. Otto vendors no database — country data goes
118
- stale, and a public-domain file you refresh on your own schedule keeps
119
- licensing and freshness in your control.
120
-
121
- Download it (IPv4+IPv6) into a path of your choosing:
122
-
123
- ```bash
124
- mkdir -p data
125
- curl -fsSL -o data/geo-whois-asn-country.mmdb \
126
- https://github.com/sapics/ip-location-db/releases/download/latest/geo-whois-asn-country.mmdb
127
- ```
128
-
129
- Refresh it on your own schedule (e.g. a daily cron job running the same curl).
130
- Any MMDB country database works — GeoLite2-Country, DB-IP Country Lite,
131
- iplocate, etc. — since `GeoResolver` tolerates the record shapes country
132
- databases actually use: nested `country.iso_code` (GeoLite2-Country style), a
133
- flat `country_code` string, and a bare-string `country`.
134
-
135
- > **Note on GeoLite2:** its EULA requires a MaxMind account/license key and
136
- > obliges consumers to refresh within 30 days of each release. A PDDL dataset
137
- > avoids both obligations.
138
-
139
- ## Header trust and spoofing
140
-
141
- Every geo header is trivially client-spoofable unless the request actually
142
- arrived through the CDN that sets it. Otto trusts geo headers — both the
143
- configured `geo_header` and the provider headers — **only** for a request that
144
- demonstrably arrived via a configured **CIDR trusted proxy**
145
- (`env['otto.via_trusted_proxy']` with `trusted_proxies` configured). A spoofed
146
- header on a direct connection is ignored, and resolution falls through to the
147
- custom resolver / database.
148
-
149
- Origins Otto cannot verify are **not** trusted:
150
-
151
- - **No trusted-proxy configuration.** A direct internet client could otherwise
152
- pick its own country by sending `CF-IPCountry` / `X-Client-Country`, so with
153
- no `trusted_proxies` configured, header steps are skipped and resolution falls
154
- to the resolver / database (`'**'` if neither is set).
155
- - **Count-based `trusted_proxy_depth` mode.** The header-setting hop cannot be
156
- verified as a geo-CDN, so depth mode does not enable header trust. This
157
- conflict fails loud: configuring a `geo_header` together with a
158
- `trusted_proxy_depth` raises `ArgumentError` at configuration time (in
159
- either order) instead of silently ignoring the header per-request.
160
- Database-backed geo remains fully supported under depth. The built-in
161
- provider headers stay legal (there is nothing to configure, so nothing
162
- can raise) but are equally inert — header trust requires CIDR-verified
163
- proxies — so only an explicitly configured `geo_header` is rejected, and
164
- depth deployments that want geo should set `geo_db_path`.
165
-
166
- **Migration:** to keep header-based geo, configure `trusted_proxies` (CIDR
167
- matchers) so Otto can verify the proxy origin. Depth-mode and header-only
168
- deployments should set `geo_db_path` for a local database instead; otherwise
169
- resolution returns `'**'`.
170
-
171
- ## Acceptance behavior summary
172
-
173
- | Scenario | Result |
174
- | --- | --- |
175
- | Configured `geo_header` present and trusted | wins over provider headers |
176
- | Request not via a verified CIDR trusted proxy | geo headers skipped |
177
- | No `trusted_proxies` configured | geo headers skipped (not trusted) |
178
- | Database lookup | uses the masked IP only |
179
- | `geo: false` | `nil`, no database in memory |
180
- | Bad `geo_db_path` | raises at boot, not per-request |
181
- | Nothing matches | `'**'` |