visionary-sdk 1.3.0 → 1.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +9 -0
- data/README.md +38 -0
- data/lib/visionary/agent.rb +11 -1
- data/lib/visionary/config.rb +4 -1
- data/lib/visionary/middleware.rb +19 -4
- data/lib/visionary/railtie.rb +18 -0
- data/lib/visionary/rate_limit/config.rb +117 -0
- data/lib/visionary/rate_limit/limiter.rb +480 -0
- data/lib/visionary/rate_limit/poller.rb +150 -0
- data/lib/visionary/rate_limit.rb +134 -0
- data/lib/visionary/version.rb +1 -1
- data/lib/visionary.rb +16 -0
- metadata +6 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 67690a306009586012a24eb4772606762aa252194f7bf0533def0d156fb9770f
|
|
4
|
+
data.tar.gz: 5c9c48e23f85a5f33ae9817a98d7c90c07574e02487808c31f8aab3715d72017
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: e80bc1ee878c4a1dcefec0efd9b27b4d1222ee2f50a5e70e6c3dca24d00845c695a34c8ee4d9573d5a98d084b36d6078955e202ec0fa05a5f4c0c9b3b73ece25
|
|
7
|
+
data.tar.gz: 8c87a26c4bc4e2e4566af2f9bb6a4418054b688f6d6f0e0c716bf93bd58d1c86f4209babdc2d89ae4136d505e9ae8dc16d8e786733c795e09da9c4cebf09b745
|
data/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,15 @@
|
|
|
2
2
|
|
|
3
3
|
## Unreleased
|
|
4
4
|
|
|
5
|
+
- Rate limits from Security → Rate limits: the middleware reads the rules
|
|
6
|
+
every 15 s and answers limited (`429` with `Retry-After`) or blocked
|
|
7
|
+
(`403`) requests before the app runs. Rules on the user, role or an
|
|
8
|
+
attribute run when the app calls `Visionary.check_rate_limit!(attrs)`
|
|
9
|
+
after `set_user`; it raises `Visionary::RateLimited`, which the middleware
|
|
10
|
+
(and in Rails a `rescue_from` from the Railtie) turns into the response.
|
|
11
|
+
Counters are per process. Outcomes go to `rate_limit_events`.
|
|
12
|
+
`VISIONARY_RATE_LIMIT=false` turns it off.
|
|
13
|
+
|
|
5
14
|
- Requests on the Settings → Data skip list (`requests.skip`) are handled
|
|
6
15
|
like paths outside `VISIONARY_LOG_PATH_PREFIX`: no `api_logs` row or spans.
|
|
7
16
|
|
data/README.md
CHANGED
|
@@ -485,6 +485,43 @@ of profiling whatever the dashboard says, set `VISIONARY_PROFILE=false`.
|
|
|
485
485
|
`routes` (the milliseconds per `METHOD route` for requests that finished
|
|
486
486
|
during the profile). See [How profiling works](#how-profiling-works).
|
|
487
487
|
|
|
488
|
+
### Rate limits
|
|
489
|
+
|
|
490
|
+
**When:** you made rules in the dashboard (Security → Rate limits) and want
|
|
491
|
+
this service to enforce them.
|
|
492
|
+
|
|
493
|
+
Nothing to add for rules on IP, country, method, path, headers or query
|
|
494
|
+
parameters: the middleware reads the rules every 15 seconds and checks each
|
|
495
|
+
request before the app runs. A limited request gets `429` with
|
|
496
|
+
`Retry-After`, a blocked one `403`; the app is not called and the request is
|
|
497
|
+
still logged. Rules keyed on the user, role or an attribute (`attr:plan`)
|
|
498
|
+
need the user, so call the check after `set_user`:
|
|
499
|
+
|
|
500
|
+
```ruby
|
|
501
|
+
# Rails: app/controllers/application_controller.rb
|
|
502
|
+
before_action do
|
|
503
|
+
if current_user
|
|
504
|
+
Visionary.set_user(current_user.id, role: current_user.role)
|
|
505
|
+
Visionary.check_rate_limit!(plan: current_user.plan)
|
|
506
|
+
end
|
|
507
|
+
end
|
|
508
|
+
```
|
|
509
|
+
|
|
510
|
+
`check_rate_limit!` raises `Visionary::RateLimited` (with `status`,
|
|
511
|
+
`headers` and `body`) when a rule limits the request. The Rack middleware
|
|
512
|
+
turns it into the response, and in Rails the Railtie adds a `rescue_from`
|
|
513
|
+
to controllers that does the same (an app-wide `rescue_from StandardError`
|
|
514
|
+
added later would catch it first). Only the first call per request counts.
|
|
515
|
+
|
|
516
|
+
Counters live in each process: with Puma or Unicorn workers (or several
|
|
517
|
+
instances) each one counts on its own, so a limit of 100 a minute on 4
|
|
518
|
+
workers lets up to 400 through. Set limits per process. Rate limiting fails
|
|
519
|
+
open: an error in it never fails a request. `VISIONARY_RATE_LIMIT=false`
|
|
520
|
+
turns enforcement and polling off.
|
|
521
|
+
|
|
522
|
+
**Records:** limited and blocked requests (and what `monitor` rules would
|
|
523
|
+
have done), aggregated, in `rate_limit_events` every 15 seconds.
|
|
524
|
+
|
|
488
525
|
### Query plans (Explain)
|
|
489
526
|
|
|
490
527
|
The Ruby SDK has no explainer. It does not answer **Explain** on the Queries
|
|
@@ -556,6 +593,7 @@ options in code, plus the hooks below. Set them before the first request
|
|
|
556
593
|
| `VISIONARY_METRICS` | `metrics` | `true` | Record custom metrics; `false` makes `counter`, `gauge` and `histogram` no-ops |
|
|
557
594
|
| `VISIONARY_METRICS_INTERVAL` | `metrics_interval` (seconds) | `15s` | How often each series is written |
|
|
558
595
|
| `VISIONARY_METRICS_MAX_SERIES` | `metrics_max_series` | `1000` | Label sets kept per metric name |
|
|
596
|
+
| `VISIONARY_RATE_LIMIT` | `rate_limit` | `true` | Enforce the dashboard's rate limits (see [Rate limits](#rate-limits)); `false` turns enforcement and polling off |
|
|
559
597
|
|
|
560
598
|
If `VISIONARY_RELEASE` is unset, the release is read from the first of
|
|
561
599
|
`RAILWAY_GIT_COMMIT_SHA`, `RENDER_GIT_COMMIT`, `VERCEL_GIT_COMMIT_SHA`,
|
data/lib/visionary/agent.rb
CHANGED
|
@@ -29,6 +29,9 @@ module Visionary
|
|
|
29
29
|
class Agent
|
|
30
30
|
attr_reader :config, :http, :telemetry, :logger, :instance, :logs
|
|
31
31
|
|
|
32
|
+
# The rate limiter, or nil when VISIONARY_RATE_LIMIT is off or there is no database.
|
|
33
|
+
def rate_limiter = @rate_limit&.limiter
|
|
34
|
+
|
|
32
35
|
def initialize(cfg)
|
|
33
36
|
@config = cfg
|
|
34
37
|
@logger = cfg.logger || StderrLogger.new
|
|
@@ -49,6 +52,10 @@ module Visionary
|
|
|
49
52
|
@telemetry = Telemetry::Recorder.new(cfg, @instance, @writer) if cfg.spans || cfg.errors
|
|
50
53
|
@logs = Logs::Pipeline.new(cfg, @instance, @store, @schemas, @logger) if cfg.logs
|
|
51
54
|
@metrics = Metrics::Recorder.new(cfg, @instance, @store, @schemas, @logger) if cfg.metrics
|
|
55
|
+
if cfg.rate_limit
|
|
56
|
+
limiter = RateLimit::Limiter.new(cfg.service, redact_headers: cfg.redact_headers)
|
|
57
|
+
@rate_limit = RateLimit::Poller.new(limiter, @store, cfg, @instance, @logger)
|
|
58
|
+
end
|
|
52
59
|
if cfg.system_metrics && cfg.system_interval.to_f.positive?
|
|
53
60
|
@system = System::Recorder.new(cfg, @instance, @store, @schemas, @http, @logger)
|
|
54
61
|
end
|
|
@@ -65,7 +72,7 @@ module Visionary
|
|
|
65
72
|
@retention = Retention.new(@store, @logger, tables)
|
|
66
73
|
end
|
|
67
74
|
|
|
68
|
-
def enabled? = !@store.nil? && !@closed && (!@writer.nil? || !@system.nil? || !@logs.nil? || !@metrics.nil?)
|
|
75
|
+
def enabled? = !@store.nil? && !@closed && (!@writer.nil? || !@system.nil? || !@logs.nil? || !@metrics.nil? || !@rate_limit.nil?)
|
|
69
76
|
|
|
70
77
|
# A skipped path is treated exactly like one outside the prefix.
|
|
71
78
|
def log_path?(path)
|
|
@@ -110,6 +117,7 @@ module Visionary
|
|
|
110
117
|
@logs&.start
|
|
111
118
|
@metrics&.start
|
|
112
119
|
@skip&.start
|
|
120
|
+
@rate_limit&.start
|
|
113
121
|
@retention.start
|
|
114
122
|
@profiler&.start
|
|
115
123
|
install_hooks
|
|
@@ -127,6 +135,7 @@ module Visionary
|
|
|
127
135
|
deadline = mono + timeout
|
|
128
136
|
@retention&.close
|
|
129
137
|
@skip&.close
|
|
138
|
+
@rate_limit&.close([timeout / 2.0, 5].min)
|
|
130
139
|
@profiler&.close
|
|
131
140
|
idle = @writer ? @writer.close(timeout) : true
|
|
132
141
|
idle = @logs.close([deadline - mono, 0].max) && idle if @logs
|
|
@@ -152,6 +161,7 @@ module Visionary
|
|
|
152
161
|
@logs&.after_fork
|
|
153
162
|
@metrics&.after_fork
|
|
154
163
|
@skip&.after_fork
|
|
164
|
+
@rate_limit&.after_fork
|
|
155
165
|
@retention&.after_fork if @retention&.needed?
|
|
156
166
|
@profiler&.after_fork
|
|
157
167
|
@logger.info("visionary: restarted in forked process #{@pid}")
|
data/lib/visionary/config.rb
CHANGED
|
@@ -23,7 +23,7 @@ module Visionary
|
|
|
23
23
|
:system_metrics, :system_interval, :system_retention_days, :disk_path,
|
|
24
24
|
:service, :version, :spans, :span_threshold_ms, :span_max, :errors,
|
|
25
25
|
:profile, :logs, :log_level, :log_queue_size,
|
|
26
|
-
:metrics, :metrics_interval, :metrics_max_series
|
|
26
|
+
:metrics, :metrics_interval, :metrics_max_series, :rate_limit
|
|
27
27
|
|
|
28
28
|
# Runs on the writer thread, once per distinct user per batch, for users
|
|
29
29
|
# missing a handle or role: ->(id) { [handle, role] } or nil.
|
|
@@ -75,6 +75,8 @@ module Visionary
|
|
|
75
75
|
@metrics = true
|
|
76
76
|
@metrics_interval = 15.0
|
|
77
77
|
@metrics_max_series = 1000
|
|
78
|
+
# false: never enforce or poll the dashboard's rate limits (VISIONARY_RATE_LIMIT).
|
|
79
|
+
@rate_limit = true
|
|
78
80
|
end
|
|
79
81
|
|
|
80
82
|
def self.from_env(env = ENV)
|
|
@@ -111,6 +113,7 @@ module Visionary
|
|
|
111
113
|
c.metrics = parse_bool(s.("VISIONARY_METRICS"), true)
|
|
112
114
|
c.metrics_interval = parse_duration(s.("VISIONARY_METRICS_INTERVAL"), 15.0)
|
|
113
115
|
c.metrics_max_series = parse_int(s.("VISIONARY_METRICS_MAX_SERIES"), 1000)
|
|
116
|
+
c.rate_limit = parse_bool(s.("VISIONARY_RATE_LIMIT"), true)
|
|
114
117
|
c
|
|
115
118
|
end
|
|
116
119
|
|
data/lib/visionary/middleware.rb
CHANGED
|
@@ -29,7 +29,7 @@ module Visionary
|
|
|
29
29
|
# Unlogged requests still pass the trace on to their outbound calls
|
|
30
30
|
span, parent = TraceContext.request_span(env)
|
|
31
31
|
TraceContext.with(span) do
|
|
32
|
-
next passthrough(agent, env) unless agent.log_path?(path)
|
|
32
|
+
next passthrough(agent, env, path) unless agent.log_path?(path)
|
|
33
33
|
|
|
34
34
|
log(agent, env, path, span, parent)
|
|
35
35
|
end
|
|
@@ -73,13 +73,13 @@ module Visionary
|
|
|
73
73
|
private
|
|
74
74
|
|
|
75
75
|
# Not logged: count it, and still record what escapes.
|
|
76
|
-
def passthrough(agent, env)
|
|
76
|
+
def passthrough(agent, env, path)
|
|
77
77
|
agent.http.started
|
|
78
78
|
Profiler::Tally.started!
|
|
79
79
|
t0 = Telemetry.now_ms
|
|
80
80
|
status = 500
|
|
81
81
|
begin
|
|
82
|
-
res =
|
|
82
|
+
res = serve(agent, env, path)
|
|
83
83
|
status = res[0].to_i
|
|
84
84
|
res
|
|
85
85
|
rescue Exception => e # re-raised below; the app's handling is unchanged
|
|
@@ -118,7 +118,7 @@ module Visionary
|
|
|
118
118
|
outer = RequestContext.current
|
|
119
119
|
RequestContext.current = ctx
|
|
120
120
|
begin
|
|
121
|
-
status, res_headers, body =
|
|
121
|
+
status, res_headers, body = serve(agent, env, path)
|
|
122
122
|
rescue Exception => e # re-raised below; the app's handling is unchanged
|
|
123
123
|
ctx.failed!
|
|
124
124
|
agent.telemetry&.exception(e, handled: false, ctx: ctx)
|
|
@@ -145,6 +145,21 @@ module Visionary
|
|
|
145
145
|
[status, res_headers, proxy]
|
|
146
146
|
end
|
|
147
147
|
|
|
148
|
+
# Runs the rate limits' request phase, then the app unless it was denied.
|
|
149
|
+
def serve(agent, env, path)
|
|
150
|
+
limiter = agent.respond_to?(:rate_limiter) ? agent.rate_limiter : nil
|
|
151
|
+
check = RateLimit::Check.start(limiter, env, path, client_ip(env))
|
|
152
|
+
outer = RateLimit::Check.current
|
|
153
|
+
RateLimit::Check.current = check
|
|
154
|
+
return RateLimit.response(check.denial) if check&.denial
|
|
155
|
+
|
|
156
|
+
@app.call(env)
|
|
157
|
+
rescue RateLimited => e
|
|
158
|
+
e.to_rack
|
|
159
|
+
ensure
|
|
160
|
+
RateLimit::Check.current = outer
|
|
161
|
+
end
|
|
162
|
+
|
|
148
163
|
def finish(agent, env, ctx, headers, req_capture, status, res_headers, resp_capture, t0)
|
|
149
164
|
agent.http.finished(status)
|
|
150
165
|
ctx.logged!
|
data/lib/visionary/railtie.rb
CHANGED
|
@@ -29,6 +29,24 @@ module Visionary
|
|
|
29
29
|
end
|
|
30
30
|
end
|
|
31
31
|
|
|
32
|
+
# ShowExceptions would turn Visionary::RateLimited into an error page
|
|
33
|
+
# before the middleware sees it, so controllers answer it themselves (the
|
|
34
|
+
# status mapping covers code outside controllers).
|
|
35
|
+
if config.respond_to?(:action_dispatch)
|
|
36
|
+
config.action_dispatch.rescue_responses["Visionary::RateLimited"] = :too_many_requests
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
initializer "visionary.rate_limit" do
|
|
40
|
+
ActiveSupport.on_load(:action_controller) do
|
|
41
|
+
rescue_from(Visionary::RateLimited) { |e| Visionary::Railtie.render_rate_limited(self, e) }
|
|
42
|
+
end
|
|
43
|
+
end
|
|
44
|
+
|
|
45
|
+
def self.render_rate_limited(controller, error)
|
|
46
|
+
error.headers.each { |k, v| controller.response.headers[k] = v unless k == "content-type" }
|
|
47
|
+
controller.render(body: error.body, status: error.status, content_type: error.headers["content-type"])
|
|
48
|
+
end
|
|
49
|
+
|
|
32
50
|
config.after_initialize do
|
|
33
51
|
cfg = Visionary.config
|
|
34
52
|
cfg.logger ||= Rails.logger
|
|
@@ -0,0 +1,117 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "json"
|
|
4
|
+
require "ipaddr"
|
|
5
|
+
|
|
6
|
+
module Visionary
|
|
7
|
+
# Rate-limit rules made in the dashboard (Security → Rate limits), enforced
|
|
8
|
+
# in the middleware. A port of sdks/go/ratelimit; see "Rate limits" in
|
|
9
|
+
# sdks/SPEC.md.
|
|
10
|
+
module RateLimit
|
|
11
|
+
SETTING_VERSION = "ratelimit.version"
|
|
12
|
+
SETTING_CONFIG = "ratelimit.config"
|
|
13
|
+
|
|
14
|
+
MAX_WINDOW = 86_400
|
|
15
|
+
MAX_COUNTERS = 10_000
|
|
16
|
+
MAX_AGGREGATES = 1000
|
|
17
|
+
|
|
18
|
+
DEFAULT_COUNTRY_HEADERS = %w[
|
|
19
|
+
CF-IPCountry CloudFront-Viewer-Country X-Vercel-IP-Country Fastly-Geo-Country X-AppEngine-Country X-Country-Code
|
|
20
|
+
].freeze
|
|
21
|
+
|
|
22
|
+
ID_PATTERN = /\A[A-Za-z0-9_-]{1,32}\z/
|
|
23
|
+
|
|
24
|
+
Config = Struct.new(:enabled, :ip_header, :country_headers, :rules, keyword_init: true) do
|
|
25
|
+
def on? = enabled != false
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
# One rule as stored; nil and zero values mean the field is missing.
|
|
29
|
+
Rule = Struct.new(:id, :name, :enabled, :mode, :action, :services, :http_methods, :paths, :ips, :countries, :match,
|
|
30
|
+
:key, :limit, :window_s, :algorithm, :burst, :limit_by, :limits, :status, :body, :content_type,
|
|
31
|
+
keyword_init: true) do
|
|
32
|
+
def on? = enabled != false
|
|
33
|
+
end
|
|
34
|
+
|
|
35
|
+
STRING_FIELDS = %w[id name mode action key algorithm limit_by body content_type].freeze
|
|
36
|
+
LIST_FIELDS = %w[services methods paths ips countries].freeze
|
|
37
|
+
NUMBER_FIELDS = %w[limit window_s burst].freeze
|
|
38
|
+
|
|
39
|
+
class << self
|
|
40
|
+
# Reads a stored document leniently: a rule with a field of the wrong
|
|
41
|
+
# type is dropped, the rest are kept. Only invalid JSON raises.
|
|
42
|
+
def parse(raw)
|
|
43
|
+
doc = JSON.parse(raw.to_s)
|
|
44
|
+
return Config.new(rules: []) unless doc.is_a?(Hash)
|
|
45
|
+
|
|
46
|
+
enabled = doc["enabled"]
|
|
47
|
+
headers = doc["country_headers"]
|
|
48
|
+
rules = doc["rules"].is_a?(Array) ? doc["rules"].filter_map { |r| decode_rule(r) } : []
|
|
49
|
+
Config.new(enabled: [true, false].include?(enabled) ? enabled : nil,
|
|
50
|
+
ip_header: doc["ip_header"].is_a?(String) ? doc["ip_header"] : "",
|
|
51
|
+
country_headers: string_list?(headers) ? Array(headers) : [], rules: rules)
|
|
52
|
+
end
|
|
53
|
+
|
|
54
|
+
def valid_id?(id) = id.is_a?(String) && ID_PATTERN.match?(id)
|
|
55
|
+
|
|
56
|
+
# [IPAddr, ...] entry for an IP or CIDR, IPv4-mapped IPv6 as IPv4; nil when it does not parse.
|
|
57
|
+
def parse_prefix(s)
|
|
58
|
+
s = s.to_s.strip
|
|
59
|
+
if s.include?("/")
|
|
60
|
+
return nil unless s.match?(%r{\A[0-9A-Fa-f:.]+/\d{1,3}\z})
|
|
61
|
+
addr = IPAddr.new(s)
|
|
62
|
+
bits = addr.prefix
|
|
63
|
+
addr = IPAddr.new("#{addr.native}/#{bits - 96}") if addr.ipv4_mapped? && bits >= 96
|
|
64
|
+
addr
|
|
65
|
+
else
|
|
66
|
+
parse_addr(s)
|
|
67
|
+
end
|
|
68
|
+
rescue IPAddr::Error, ArgumentError
|
|
69
|
+
nil
|
|
70
|
+
end
|
|
71
|
+
|
|
72
|
+
def parse_addr(s)
|
|
73
|
+
s = s.to_s.strip.sub(/%.*\z/, "")
|
|
74
|
+
return nil unless s.match?(/\A[0-9A-Fa-f:.]+\z/)
|
|
75
|
+
addr = IPAddr.new(s)
|
|
76
|
+
addr.ipv4_mapped? ? IPAddr.new(addr.native.to_s) : addr
|
|
77
|
+
rescue IPAddr::Error, ArgumentError
|
|
78
|
+
nil
|
|
79
|
+
end
|
|
80
|
+
|
|
81
|
+
private
|
|
82
|
+
|
|
83
|
+
def string_list?(v) = v.nil? || (v.is_a?(Array) && v.all?(String))
|
|
84
|
+
|
|
85
|
+
def decode_rule(r)
|
|
86
|
+
return nil unless r.nil? || r.is_a?(Hash)
|
|
87
|
+
r ||= {}
|
|
88
|
+
out = {}
|
|
89
|
+
STRING_FIELDS.each do |f|
|
|
90
|
+
v = r[f]
|
|
91
|
+
return nil unless v.nil? || v.is_a?(String)
|
|
92
|
+
out[f.to_sym] = v.to_s
|
|
93
|
+
end
|
|
94
|
+
LIST_FIELDS.each do |f|
|
|
95
|
+
return nil unless string_list?(r[f])
|
|
96
|
+
out[f == "methods" ? :http_methods : f.to_sym] = Array(r[f])
|
|
97
|
+
end
|
|
98
|
+
NUMBER_FIELDS.each do |f|
|
|
99
|
+
v = r[f]
|
|
100
|
+
return nil unless v.nil? || v.is_a?(Numeric)
|
|
101
|
+
out[f.to_sym] = v.to_f
|
|
102
|
+
end
|
|
103
|
+
return nil unless [nil, true, false].include?(r["enabled"])
|
|
104
|
+
out[:enabled] = r["enabled"]
|
|
105
|
+
return nil unless r["status"].nil? || r["status"].is_a?(Integer)
|
|
106
|
+
out[:status] = r["status"].to_i
|
|
107
|
+
match = r["match"]
|
|
108
|
+
return nil unless match.nil? || (match.is_a?(Hash) && match.values.all? { |v| string_list?(v) })
|
|
109
|
+
out[:match] = (match || {}).transform_values { |v| Array(v) }
|
|
110
|
+
limits = r["limits"]
|
|
111
|
+
return nil unless limits.nil? || (limits.is_a?(Hash) && limits.values.all?(Numeric))
|
|
112
|
+
out[:limits] = (limits || {}).transform_values(&:to_f)
|
|
113
|
+
Rule.new(**out)
|
|
114
|
+
end
|
|
115
|
+
end
|
|
116
|
+
end
|
|
117
|
+
end
|
|
@@ -0,0 +1,480 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "digest"
|
|
4
|
+
|
|
5
|
+
module Visionary
|
|
6
|
+
module RateLimit
|
|
7
|
+
LIMITED = "limited"
|
|
8
|
+
BLOCKED = "blocked"
|
|
9
|
+
WOULD_LIMIT = "would_limit"
|
|
10
|
+
WOULD_BLOCK = "would_block"
|
|
11
|
+
|
|
12
|
+
USER_PROPS = %i[user role attr].freeze
|
|
13
|
+
|
|
14
|
+
# What the limiter sees of a request. header and query are callables
|
|
15
|
+
# returning a header's (lower-case name) or query parameter's first value.
|
|
16
|
+
Request = Struct.new(:http_method, :path, :remote_ip, :header, :query)
|
|
17
|
+
|
|
18
|
+
User = Struct.new(:id, :role)
|
|
19
|
+
|
|
20
|
+
# The answer for a request that is limited or blocked.
|
|
21
|
+
Denial = Struct.new(:status, :rule_id, :retry_after, :body, :content_type, keyword_init: true)
|
|
22
|
+
|
|
23
|
+
# One rate_limit_events aggregate.
|
|
24
|
+
Event = Struct.new(:created_at, :rule_id, :rule_name, :outcome, :key_value, :ip, :country, :http_method, :path,
|
|
25
|
+
:user_id, :hits, keyword_init: true)
|
|
26
|
+
|
|
27
|
+
# Carries one request between the two phases.
|
|
28
|
+
State = Struct.new(:set, :ip, :country, :allowed, :user_done, :denial)
|
|
29
|
+
|
|
30
|
+
# A property: [kind, name], name being a lower-case header, query or attribute name.
|
|
31
|
+
def self.parse_prop(s)
|
|
32
|
+
s = s.to_s
|
|
33
|
+
if s.start_with?("header:") && s.length > 7 then [:header, s[7..].downcase]
|
|
34
|
+
elsif s.start_with?("query:") && s.length > 6 then [:query, s[6..]]
|
|
35
|
+
elsif s.start_with?("attr:") && s.length > 5 then [:attr, s[5..]]
|
|
36
|
+
elsif %w[ip country method path global user role].include?(s) then [s.to_sym, ""]
|
|
37
|
+
end
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
# Matches a whole path against a pattern where * is any run of characters.
|
|
41
|
+
def self.glob(pattern, str)
|
|
42
|
+
pat = pattern.b
|
|
43
|
+
s = str.to_s.b
|
|
44
|
+
p = i = 0
|
|
45
|
+
star = -1
|
|
46
|
+
mark = 0
|
|
47
|
+
while i < s.bytesize
|
|
48
|
+
if p < pat.bytesize && pat.getbyte(p) == 42
|
|
49
|
+
star = p
|
|
50
|
+
mark = i
|
|
51
|
+
p += 1
|
|
52
|
+
elsif p < pat.bytesize && pat.getbyte(p) == s.getbyte(i)
|
|
53
|
+
p += 1
|
|
54
|
+
i += 1
|
|
55
|
+
elsif star >= 0
|
|
56
|
+
p = star + 1
|
|
57
|
+
mark += 1
|
|
58
|
+
i = mark
|
|
59
|
+
else
|
|
60
|
+
return false
|
|
61
|
+
end
|
|
62
|
+
end
|
|
63
|
+
p += 1 while p < pat.bytesize && pat.getbyte(p) == 42
|
|
64
|
+
p == pat.bytesize
|
|
65
|
+
end
|
|
66
|
+
|
|
67
|
+
def self.clip(s, n)
|
|
68
|
+
s = s.to_s
|
|
69
|
+
s.length > n ? s[0, n] : s
|
|
70
|
+
end
|
|
71
|
+
|
|
72
|
+
# Per-key counters of one rule.
|
|
73
|
+
class CounterSet
|
|
74
|
+
attr_reader :mutex, :map
|
|
75
|
+
|
|
76
|
+
def initialize
|
|
77
|
+
@mutex = Mutex.new
|
|
78
|
+
@map = {}
|
|
79
|
+
end
|
|
80
|
+
|
|
81
|
+
def clear! = @map = {}
|
|
82
|
+
end
|
|
83
|
+
|
|
84
|
+
# A counter: win/curr/prev for fixed and sliding windows, tokens/last for the bucket.
|
|
85
|
+
Counter = Struct.new(:win, :curr, :prev, :tokens, :last)
|
|
86
|
+
|
|
87
|
+
class CompiledRule
|
|
88
|
+
attr_accessor :id, :name, :monitor, :action, :verbs, :paths, :has_ips, :prefixes, :countries, :match, :key,
|
|
89
|
+
:limit, :window, :alg, :burst, :limit_by, :limits, :status, :body, :content_type, :user_phase,
|
|
90
|
+
:counters
|
|
91
|
+
|
|
92
|
+
def initialize
|
|
93
|
+
@verbs = []
|
|
94
|
+
@paths = []
|
|
95
|
+
@prefixes = []
|
|
96
|
+
@countries = []
|
|
97
|
+
@match = []
|
|
98
|
+
@limits = {}
|
|
99
|
+
@user_phase = false
|
|
100
|
+
@limit = @window = @burst = 0.0
|
|
101
|
+
@alg = :sliding
|
|
102
|
+
end
|
|
103
|
+
|
|
104
|
+
# What the counters depend on; a change resets them.
|
|
105
|
+
def shape = [key, window, alg, burst]
|
|
106
|
+
|
|
107
|
+
def deny(retry_after)
|
|
108
|
+
st = status.to_i
|
|
109
|
+
st = action == "block" ? 403 : 429 if st.zero?
|
|
110
|
+
if !body.empty?
|
|
111
|
+
out = body.gsub("{retry_after}", retry_after.to_s)
|
|
112
|
+
ct = body.lstrip.start_with?("{", "[") ? "application/json" : "text/plain; charset=utf-8"
|
|
113
|
+
elsif action == "block"
|
|
114
|
+
out = %({"error":{"code":"blocked","key":"#{id}","message":"Forbidden"}})
|
|
115
|
+
ct = "application/json"
|
|
116
|
+
else
|
|
117
|
+
out = %({"error":{"code":"rate_limited","key":"#{id}","message":"Too many requests"}})
|
|
118
|
+
ct = "application/json"
|
|
119
|
+
end
|
|
120
|
+
ct = content_type unless content_type.empty?
|
|
121
|
+
Denial.new(status: st, rule_id: id, retry_after: retry_after, body: out, content_type: ct)
|
|
122
|
+
end
|
|
123
|
+
|
|
124
|
+
def capacity(lim) = burst.positive? ? burst : lim
|
|
125
|
+
|
|
126
|
+
# Takes one request on key's counter: [retry_after, over].
|
|
127
|
+
def hit(k, lim, t)
|
|
128
|
+
w_s = window
|
|
129
|
+
return [w_s.ceil, true] if lim.zero?
|
|
130
|
+
|
|
131
|
+
counters.mutex.synchronize do
|
|
132
|
+
c = counters.map[k]
|
|
133
|
+
unless c
|
|
134
|
+
if counters.map.size >= MAX_COUNTERS
|
|
135
|
+
sweep_locked(t)
|
|
136
|
+
counters.clear! if counters.map.size >= MAX_COUNTERS
|
|
137
|
+
end
|
|
138
|
+
c = Counter.new(nil, 0.0, 0.0, capacity(lim), t)
|
|
139
|
+
counters.map[k] = c
|
|
140
|
+
end
|
|
141
|
+
if alg == :token_bucket
|
|
142
|
+
cap = capacity(lim)
|
|
143
|
+
rate = lim / w_s
|
|
144
|
+
if t > c.last
|
|
145
|
+
c.tokens = [cap, c.tokens + ((t - c.last) * rate)].min
|
|
146
|
+
c.last = t
|
|
147
|
+
end
|
|
148
|
+
return [[1, ((1 - c.tokens) / rate).ceil].max, true] if c.tokens < 1
|
|
149
|
+
c.tokens -= 1
|
|
150
|
+
return [0, false]
|
|
151
|
+
end
|
|
152
|
+
w = (t / w_s).floor
|
|
153
|
+
if c.win.nil?
|
|
154
|
+
c.prev = c.curr = 0.0
|
|
155
|
+
elsif w == c.win + 1
|
|
156
|
+
c.prev = c.curr
|
|
157
|
+
c.curr = 0.0
|
|
158
|
+
elsif w > c.win + 1
|
|
159
|
+
c.prev = c.curr = 0.0
|
|
160
|
+
end
|
|
161
|
+
c.win = c.win.nil? ? w : [c.win, w].max
|
|
162
|
+
over = if alg == :fixed
|
|
163
|
+
c.curr >= lim
|
|
164
|
+
else
|
|
165
|
+
e = t - (w * w_s)
|
|
166
|
+
(c.prev * (w_s - e) / w_s) + c.curr + 1 > lim
|
|
167
|
+
end
|
|
168
|
+
return [[1, (((w + 1) * w_s) - t).ceil].max, true] if over
|
|
169
|
+
c.curr += 1
|
|
170
|
+
[0, false]
|
|
171
|
+
end
|
|
172
|
+
end
|
|
173
|
+
|
|
174
|
+
# Drops idle counters; the caller holds the set's mutex.
|
|
175
|
+
def sweep_locked(t)
|
|
176
|
+
w = (t / window).floor
|
|
177
|
+
counters.map.delete_if do |_, c|
|
|
178
|
+
if alg == :token_bucket
|
|
179
|
+
# the limit may differ per key; the static one is close enough to call it idle
|
|
180
|
+
limit <= 0 || c.tokens + ((t - c.last) * (limit / window)) >= capacity(limit)
|
|
181
|
+
else
|
|
182
|
+
c.win < w - 1
|
|
183
|
+
end
|
|
184
|
+
end
|
|
185
|
+
end
|
|
186
|
+
end
|
|
187
|
+
|
|
188
|
+
RuleSet = Struct.new(:ip_header, :country_headers, :request, :user, :by_id)
|
|
189
|
+
|
|
190
|
+
# Evaluates the rules in force and keeps the counters. Thread-safe.
|
|
191
|
+
class Limiter
|
|
192
|
+
DEFAULT_REDACT = %w[Authorization Proxy-Authorization Cookie Set-Cookie X-Api-Key X-Auth-Token].freeze
|
|
193
|
+
|
|
194
|
+
# Sees every recorded outcome as (rule_id, outcome); for tests.
|
|
195
|
+
attr_accessor :trace
|
|
196
|
+
|
|
197
|
+
def initialize(service = nil, redact_headers: [])
|
|
198
|
+
@service = service.to_s
|
|
199
|
+
@set = nil
|
|
200
|
+
@shapes = {}
|
|
201
|
+
@load_mutex = Mutex.new
|
|
202
|
+
@agg_mutex = Mutex.new
|
|
203
|
+
@agg = {}
|
|
204
|
+
@redact = (DEFAULT_REDACT + Array(redact_headers)).to_h { |h| [h.to_s.strip.downcase, true] }
|
|
205
|
+
end
|
|
206
|
+
|
|
207
|
+
# Puts a parsed config in force, keeping the counters of rules whose
|
|
208
|
+
# key, window, algorithm and burst did not change.
|
|
209
|
+
def load(config)
|
|
210
|
+
@load_mutex.synchronize do
|
|
211
|
+
nxt = compile(config)
|
|
212
|
+
old = @set
|
|
213
|
+
shapes = {}
|
|
214
|
+
nxt.by_id.each do |id, r|
|
|
215
|
+
shapes[id] = r.shape
|
|
216
|
+
r.counters = if old && @shapes[id] == shapes[id] && old.by_id[id]
|
|
217
|
+
old.by_id[id].counters
|
|
218
|
+
else
|
|
219
|
+
CounterSet.new
|
|
220
|
+
end
|
|
221
|
+
end
|
|
222
|
+
@shapes = shapes
|
|
223
|
+
@set = nxt
|
|
224
|
+
end
|
|
225
|
+
end
|
|
226
|
+
|
|
227
|
+
def clear = load(Config.new(rules: []))
|
|
228
|
+
|
|
229
|
+
def active?
|
|
230
|
+
s = @set
|
|
231
|
+
!s.nil? && !s.by_id.empty?
|
|
232
|
+
end
|
|
233
|
+
|
|
234
|
+
# The request phase; the State goes to check_user later.
|
|
235
|
+
def check_request(req, now)
|
|
236
|
+
set = @set
|
|
237
|
+
st = State.new(nil, "", "", false, false, nil)
|
|
238
|
+
return st if set.nil? || set.by_id.empty?
|
|
239
|
+
|
|
240
|
+
st.set = set
|
|
241
|
+
st.ip = client_ip(req, set.ip_header)
|
|
242
|
+
st.country = country(req, set.country_headers)
|
|
243
|
+
run(set.request, st, req, nil, {}, now)
|
|
244
|
+
st
|
|
245
|
+
rescue StandardError
|
|
246
|
+
State.new(nil, "", "", false, false, nil) # fail open
|
|
247
|
+
end
|
|
248
|
+
|
|
249
|
+
# The user phase, once per request; later calls repeat the first outcome.
|
|
250
|
+
def check_user(st, req, user, attrs, now)
|
|
251
|
+
return nil if st.nil? || st.set.nil?
|
|
252
|
+
return st.denial if st.user_done || st.allowed || st.denial
|
|
253
|
+
|
|
254
|
+
st.user_done = true
|
|
255
|
+
return nil if st.set.user.empty?
|
|
256
|
+
|
|
257
|
+
begin
|
|
258
|
+
run(st.set.user, st, req, user || User.new("", ""), attrs || {}, now)
|
|
259
|
+
rescue StandardError
|
|
260
|
+
st.denial = nil
|
|
261
|
+
end
|
|
262
|
+
st.denial
|
|
263
|
+
end
|
|
264
|
+
|
|
265
|
+
# Drops idle counters of every rule; called at each poll.
|
|
266
|
+
def sweep(now)
|
|
267
|
+
set = @set
|
|
268
|
+
return unless set
|
|
269
|
+
set.by_id.each_value do |r|
|
|
270
|
+
next unless r.counters && r.action == "limit"
|
|
271
|
+
r.counters.mutex.synchronize { r.sweep_locked(now.to_f) }
|
|
272
|
+
end
|
|
273
|
+
end
|
|
274
|
+
|
|
275
|
+
# The outcomes recorded since the last call, oldest first.
|
|
276
|
+
def drain
|
|
277
|
+
@agg_mutex.synchronize do
|
|
278
|
+
out = @agg.values
|
|
279
|
+
@agg = {}
|
|
280
|
+
out
|
|
281
|
+
end
|
|
282
|
+
end
|
|
283
|
+
|
|
284
|
+
private
|
|
285
|
+
|
|
286
|
+
def compile(c)
|
|
287
|
+
headers = Array(c.country_headers).map { |h| h.to_s.strip }.reject(&:empty?)
|
|
288
|
+
headers = DEFAULT_COUNTRY_HEADERS if headers.empty?
|
|
289
|
+
rs = RuleSet.new(c.ip_header.to_s.strip, headers, [], [], {})
|
|
290
|
+
return rs unless c.on?
|
|
291
|
+
|
|
292
|
+
Array(c.rules).each do |src|
|
|
293
|
+
r = compile_rule(src)
|
|
294
|
+
next if r.nil? || !src.on? || rs.by_id[r.id] || !serves?(Array(src.services))
|
|
295
|
+
rs.by_id[r.id] = r
|
|
296
|
+
(r.user_phase ? rs.user : rs.request) << r
|
|
297
|
+
end
|
|
298
|
+
rs
|
|
299
|
+
end
|
|
300
|
+
|
|
301
|
+
def serves?(list)
|
|
302
|
+
return true if list.empty?
|
|
303
|
+
return false if @service.empty?
|
|
304
|
+
list.include?(@service)
|
|
305
|
+
end
|
|
306
|
+
|
|
307
|
+
def compile_rule(src)
|
|
308
|
+
return nil unless RateLimit.valid_id?(src.id)
|
|
309
|
+
r = CompiledRule.new
|
|
310
|
+
r.id = src.id
|
|
311
|
+
r.name = src.name.to_s.empty? ? src.id : src.name
|
|
312
|
+
r.action = src.action.to_s
|
|
313
|
+
r.status = src.status.to_i
|
|
314
|
+
r.body = src.body.to_s
|
|
315
|
+
r.content_type = src.content_type.to_s
|
|
316
|
+
case src.mode.to_s
|
|
317
|
+
when "", "enforce" then r.monitor = false
|
|
318
|
+
when "monitor" then r.monitor = true
|
|
319
|
+
else return nil
|
|
320
|
+
end
|
|
321
|
+
return nil unless %w[limit block allow].include?(r.action)
|
|
322
|
+
|
|
323
|
+
r.verbs = Array(src.http_methods).map { |m| m.strip.upcase }
|
|
324
|
+
r.paths = Array(src.paths)
|
|
325
|
+
r.has_ips = !Array(src.ips).empty?
|
|
326
|
+
r.prefixes = Array(src.ips).filter_map { |s| RateLimit.parse_prefix(s) }
|
|
327
|
+
r.countries = Array(src.countries).map { |c| c.strip.upcase }
|
|
328
|
+
(src.match || {}).each do |name, values|
|
|
329
|
+
p = RateLimit.parse_prop(name)
|
|
330
|
+
return nil if p.nil? || p[0] == :global
|
|
331
|
+
r.match << [p, values]
|
|
332
|
+
r.user_phase ||= USER_PROPS.include?(p[0])
|
|
333
|
+
end
|
|
334
|
+
return r unless r.action == "limit"
|
|
335
|
+
|
|
336
|
+
key = RateLimit.parse_prop(src.key.to_s.empty? ? "ip" : src.key)
|
|
337
|
+
return nil unless key
|
|
338
|
+
r.key = key
|
|
339
|
+
r.user_phase ||= USER_PROPS.include?(key[0])
|
|
340
|
+
w = src.window_s.to_f
|
|
341
|
+
return nil if w < 1 || w > MAX_WINDOW
|
|
342
|
+
r.window = w
|
|
343
|
+
r.limit = src.limit.to_f
|
|
344
|
+
r.burst = src.burst.to_f
|
|
345
|
+
r.alg = case src.algorithm.to_s
|
|
346
|
+
when "", "sliding" then :sliding
|
|
347
|
+
when "fixed" then :fixed
|
|
348
|
+
when "token_bucket" then :token_bucket
|
|
349
|
+
else return nil
|
|
350
|
+
end
|
|
351
|
+
unless src.limit_by.to_s.empty?
|
|
352
|
+
p = RateLimit.parse_prop(src.limit_by)
|
|
353
|
+
return nil if p.nil? || p[0] == :global
|
|
354
|
+
r.limit_by = p
|
|
355
|
+
r.limits = src.limits || {}
|
|
356
|
+
r.user_phase ||= USER_PROPS.include?(p[0])
|
|
357
|
+
end
|
|
358
|
+
r
|
|
359
|
+
end
|
|
360
|
+
|
|
361
|
+
def run(rules, st, req, user, attrs, now)
|
|
362
|
+
t = now.to_f
|
|
363
|
+
rules.each do |r|
|
|
364
|
+
next unless matches?(r, st, req, user, attrs)
|
|
365
|
+
case r.action
|
|
366
|
+
when "allow"
|
|
367
|
+
next if r.monitor
|
|
368
|
+
st.allowed = true
|
|
369
|
+
return
|
|
370
|
+
when "block"
|
|
371
|
+
if r.monitor
|
|
372
|
+
record(r, WOULD_BLOCK, "", st, req, user, t)
|
|
373
|
+
next
|
|
374
|
+
end
|
|
375
|
+
record(r, BLOCKED, "", st, req, user, t)
|
|
376
|
+
st.denial = r.deny(0)
|
|
377
|
+
return
|
|
378
|
+
when "limit"
|
|
379
|
+
k = value(r.key, st, req, user, attrs)
|
|
380
|
+
next if k.empty?
|
|
381
|
+
lim = r.limit
|
|
382
|
+
if r.limit_by
|
|
383
|
+
n = r.limits[value(r.limit_by, st, req, user, attrs)]
|
|
384
|
+
lim = n unless n.nil?
|
|
385
|
+
end
|
|
386
|
+
next if lim.negative?
|
|
387
|
+
retry_after, over = r.hit(k, lim, t)
|
|
388
|
+
next unless over
|
|
389
|
+
if r.monitor
|
|
390
|
+
record(r, WOULD_LIMIT, k, st, req, user, t)
|
|
391
|
+
next
|
|
392
|
+
end
|
|
393
|
+
record(r, LIMITED, k, st, req, user, t)
|
|
394
|
+
st.denial = r.deny(retry_after)
|
|
395
|
+
return
|
|
396
|
+
end
|
|
397
|
+
end
|
|
398
|
+
end
|
|
399
|
+
|
|
400
|
+
def matches?(r, st, req, user, attrs)
|
|
401
|
+
return false if !r.verbs.empty? && !r.verbs.include?(req.http_method.to_s.upcase)
|
|
402
|
+
return false if !r.paths.empty? && r.paths.none? { |g| RateLimit.glob(g, req.path) }
|
|
403
|
+
return false if r.has_ips && !in_prefixes?(r.prefixes, st.ip)
|
|
404
|
+
return false if !r.countries.empty? && (st.country.empty? || !r.countries.include?(st.country))
|
|
405
|
+
|
|
406
|
+
r.match.all? do |p, values|
|
|
407
|
+
v = value(p, st, req, user, attrs)
|
|
408
|
+
values.empty? ? !v.empty? : values.include?(v)
|
|
409
|
+
end
|
|
410
|
+
end
|
|
411
|
+
|
|
412
|
+
def value(prop, st, req, user, attrs)
|
|
413
|
+
kind, name = prop
|
|
414
|
+
v = case kind
|
|
415
|
+
when :ip then st.ip
|
|
416
|
+
when :country then st.country
|
|
417
|
+
when :method then req.http_method.to_s.upcase
|
|
418
|
+
when :path then req.path
|
|
419
|
+
when :header then req.header&.call(name)
|
|
420
|
+
when :query then req.query&.call(name)
|
|
421
|
+
when :global then "*"
|
|
422
|
+
when :user then user&.id
|
|
423
|
+
when :role then user&.role
|
|
424
|
+
when :attr then attrs[name]
|
|
425
|
+
end
|
|
426
|
+
v.to_s
|
|
427
|
+
end
|
|
428
|
+
|
|
429
|
+
def record(r, outcome, key, st, req, user, t)
|
|
430
|
+
@trace&.call(r.id, outcome)
|
|
431
|
+
if !key.empty? && r.key[0] == :header && @redact[r.key[1]]
|
|
432
|
+
key = "sha256:#{Digest::SHA256.hexdigest(key)[0, 12]}"
|
|
433
|
+
end
|
|
434
|
+
key = RateLimit.clip(key, 200)
|
|
435
|
+
ip = st.ip
|
|
436
|
+
@agg_mutex.synchronize do
|
|
437
|
+
id = [r.id, outcome, key, ip]
|
|
438
|
+
e = @agg[id]
|
|
439
|
+
if e.nil? && @agg.size >= MAX_AGGREGATES
|
|
440
|
+
key = "(other)"
|
|
441
|
+
ip = ""
|
|
442
|
+
id = [r.id, outcome, key, ip]
|
|
443
|
+
e = @agg[id]
|
|
444
|
+
end
|
|
445
|
+
unless e
|
|
446
|
+
e = Event.new(created_at: Time.at(t), rule_id: r.id, rule_name: r.name, outcome: outcome, key_value: key,
|
|
447
|
+
ip: ip, country: st.country, http_method: req.http_method.to_s.upcase,
|
|
448
|
+
path: RateLimit.clip(req.path, 300), user_id: user ? user.id.to_s : "", hits: 0)
|
|
449
|
+
@agg[id] = e
|
|
450
|
+
end
|
|
451
|
+
e.hits += 1
|
|
452
|
+
end
|
|
453
|
+
end
|
|
454
|
+
|
|
455
|
+
def client_ip(req, header)
|
|
456
|
+
if !header.empty? && req.header
|
|
457
|
+
v = req.header.call(header.downcase).to_s
|
|
458
|
+
first = v.split(",", 2).first.to_s.strip
|
|
459
|
+
return first unless first.empty?
|
|
460
|
+
end
|
|
461
|
+
req.remote_ip.to_s
|
|
462
|
+
end
|
|
463
|
+
|
|
464
|
+
def country(req, headers)
|
|
465
|
+
return "" unless req.header
|
|
466
|
+
headers.each do |h|
|
|
467
|
+
v = req.header.call(h.downcase).to_s.strip
|
|
468
|
+
return v.upcase unless v.empty?
|
|
469
|
+
end
|
|
470
|
+
""
|
|
471
|
+
end
|
|
472
|
+
|
|
473
|
+
def in_prefixes?(list, ip)
|
|
474
|
+
addr = RateLimit.parse_addr(ip)
|
|
475
|
+
return false unless addr
|
|
476
|
+
list.any? { |p| p.family == addr.family && p.include?(addr) }
|
|
477
|
+
end
|
|
478
|
+
end
|
|
479
|
+
end
|
|
480
|
+
end
|
|
@@ -0,0 +1,150 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Visionary
|
|
4
|
+
module RateLimit
|
|
5
|
+
# Reads the rules from visionary_settings at start and every 15 s, and
|
|
6
|
+
# writes the aggregated outcomes to rate_limit_events at each poll and
|
|
7
|
+
# at close. A failed read keeps the rules in force.
|
|
8
|
+
class Poller
|
|
9
|
+
POLL_S = 15
|
|
10
|
+
D1_BATCH = 50
|
|
11
|
+
SQL_ROWS = 500
|
|
12
|
+
VERSION_SQL = "SELECT value FROM visionary_settings WHERE name = '#{SETTING_VERSION}'".freeze
|
|
13
|
+
CONFIG_SQL = "SELECT value FROM visionary_settings WHERE name = '#{SETTING_CONFIG}'".freeze
|
|
14
|
+
COLUMNS = %w[created_at service instance rule_id rule_name outcome key_value ip country method path user_id
|
|
15
|
+
hits].freeze
|
|
16
|
+
|
|
17
|
+
attr_reader :limiter
|
|
18
|
+
|
|
19
|
+
def initialize(limiter, store, cfg, instance, logger)
|
|
20
|
+
@limiter = limiter
|
|
21
|
+
@store = store
|
|
22
|
+
@service = cfg.service
|
|
23
|
+
@instance = instance
|
|
24
|
+
@logger = logger
|
|
25
|
+
@version = ""
|
|
26
|
+
@bad_version = nil
|
|
27
|
+
@write_failed = false
|
|
28
|
+
reset
|
|
29
|
+
end
|
|
30
|
+
|
|
31
|
+
# One round: rules, idle counters, outcomes.
|
|
32
|
+
def poll(now = Time.now)
|
|
33
|
+
sync_rules
|
|
34
|
+
@limiter.sweep(now.to_f)
|
|
35
|
+
flush
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
def sync_rules
|
|
39
|
+
rows = @store.rows(VERSION_SQL)
|
|
40
|
+
version = rows.first&.first
|
|
41
|
+
if version.nil?
|
|
42
|
+
@limiter.clear unless @version.empty?
|
|
43
|
+
@version = ""
|
|
44
|
+
return
|
|
45
|
+
end
|
|
46
|
+
version = version.to_s
|
|
47
|
+
return if version == @version
|
|
48
|
+
|
|
49
|
+
raw = @store.rows(CONFIG_SQL).first&.first.to_s
|
|
50
|
+
begin
|
|
51
|
+
config = raw.empty? ? Config.new(rules: []) : RateLimit.parse(raw)
|
|
52
|
+
rescue JSON::ParserError => e
|
|
53
|
+
if @bad_version != version
|
|
54
|
+
@bad_version = version
|
|
55
|
+
@logger.warn("visionary: rate limit rules are not valid JSON; keeping the previous ones (#{e.message})")
|
|
56
|
+
end
|
|
57
|
+
return
|
|
58
|
+
end
|
|
59
|
+
@limiter.load(config)
|
|
60
|
+
@version = version
|
|
61
|
+
rescue StandardError
|
|
62
|
+
nil # keep the rules in force
|
|
63
|
+
end
|
|
64
|
+
|
|
65
|
+
def flush
|
|
66
|
+
events = @limiter.drain
|
|
67
|
+
return if events.empty?
|
|
68
|
+
|
|
69
|
+
rows = events.map do |e|
|
|
70
|
+
[Visionary.timestamp(e.created_at), @service, @instance, e.rule_id, e.rule_name, e.outcome, nz(e.key_value),
|
|
71
|
+
nz(e.ip), nz(e.country), nz(e.http_method), nz(e.path), nz(e.user_id), e.hits]
|
|
72
|
+
end
|
|
73
|
+
statements(rows).each_slice(D1_BATCH) { |chunk| @store.batch(chunk) }
|
|
74
|
+
rescue StandardError => e
|
|
75
|
+
unless @write_failed
|
|
76
|
+
@write_failed = true
|
|
77
|
+
@logger.info("visionary: could not record rate-limited requests " \
|
|
78
|
+
"(the Visionary server creates rate_limit_events): #{e.message}")
|
|
79
|
+
end
|
|
80
|
+
end
|
|
81
|
+
|
|
82
|
+
def start
|
|
83
|
+
@thread = Thread.new { run }
|
|
84
|
+
@thread.name = "visionary-ratelimit" if @thread.respond_to?(:name=)
|
|
85
|
+
self
|
|
86
|
+
end
|
|
87
|
+
|
|
88
|
+
# The parent's outcomes are its own to write.
|
|
89
|
+
def after_fork
|
|
90
|
+
@limiter.drain
|
|
91
|
+
reset
|
|
92
|
+
start
|
|
93
|
+
end
|
|
94
|
+
|
|
95
|
+
# Stops the thread, which writes what is left, waiting at most timeout seconds.
|
|
96
|
+
def close(timeout = 5)
|
|
97
|
+
@mutex.synchronize do
|
|
98
|
+
@stopped = true
|
|
99
|
+
@cond.broadcast
|
|
100
|
+
end
|
|
101
|
+
if @thread
|
|
102
|
+
@thread.join(timeout)
|
|
103
|
+
else
|
|
104
|
+
flush
|
|
105
|
+
end
|
|
106
|
+
end
|
|
107
|
+
|
|
108
|
+
private
|
|
109
|
+
|
|
110
|
+
def statements(rows)
|
|
111
|
+
if @store.string_params?
|
|
112
|
+
sql = "INSERT INTO rate_limit_events (#{COLUMNS.join(', ')}) VALUES (#{(["NULLIF(?, '')"] * COLUMNS.size).join(', ')})"
|
|
113
|
+
return rows.map { |r| [sql, r] }
|
|
114
|
+
end
|
|
115
|
+
pg = @store.respond_to?(:dialect) && @store.dialect == "postgres"
|
|
116
|
+
rows.each_slice(SQL_ROWS).map do |slice|
|
|
117
|
+
one = "(#{(['?'] * COLUMNS.size).join(', ')})"
|
|
118
|
+
sql = "INSERT INTO rate_limit_events (#{COLUMNS.join(', ')}) VALUES #{([one] * slice.size).join(', ')}"
|
|
119
|
+
args = slice.flatten(1)
|
|
120
|
+
args = args.map { |v| v.is_a?(String) && v.include?("\0") ? v.delete("\0") : v } if pg
|
|
121
|
+
[sql, args]
|
|
122
|
+
end
|
|
123
|
+
end
|
|
124
|
+
|
|
125
|
+
def nz(s) = s.nil? || s.to_s.empty? ? nil : s
|
|
126
|
+
|
|
127
|
+
def reset
|
|
128
|
+
@mutex = Mutex.new
|
|
129
|
+
@cond = ConditionVariable.new
|
|
130
|
+
@stopped = false
|
|
131
|
+
@thread = nil
|
|
132
|
+
end
|
|
133
|
+
|
|
134
|
+
def run
|
|
135
|
+
Telemetry.internal!
|
|
136
|
+
loop do
|
|
137
|
+
poll
|
|
138
|
+
stopped = @mutex.synchronize do
|
|
139
|
+
@cond.wait(@mutex, POLL_S) unless @stopped
|
|
140
|
+
@stopped
|
|
141
|
+
end
|
|
142
|
+
break if stopped
|
|
143
|
+
end
|
|
144
|
+
flush
|
|
145
|
+
rescue StandardError => e
|
|
146
|
+
@logger.error("visionary: rate limits stopped: #{e.message}")
|
|
147
|
+
end
|
|
148
|
+
end
|
|
149
|
+
end
|
|
150
|
+
end
|
|
@@ -0,0 +1,134 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "uri"
|
|
4
|
+
require_relative "rate_limit/config"
|
|
5
|
+
require_relative "rate_limit/limiter"
|
|
6
|
+
require_relative "rate_limit/poller"
|
|
7
|
+
|
|
8
|
+
module Visionary
|
|
9
|
+
# Raised by Visionary.check_rate_limit! when a rule limits or blocks the
|
|
10
|
+
# request. The Rack middleware (and in Rails a rescue_from the Railtie adds)
|
|
11
|
+
# turns it into the response.
|
|
12
|
+
class RateLimited < StandardError
|
|
13
|
+
attr_reader :status, :headers, :body
|
|
14
|
+
|
|
15
|
+
def initialize(denial)
|
|
16
|
+
@status = denial.status
|
|
17
|
+
@headers = RateLimit.headers(denial)
|
|
18
|
+
@body = denial.body
|
|
19
|
+
super("rate limited by rule #{denial.rule_id}")
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
def to_rack = [status, headers.dup, [body]]
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
module RateLimit
|
|
26
|
+
def self.headers(denial)
|
|
27
|
+
h = { "content-type" => denial.content_type }
|
|
28
|
+
h["retry-after"] = denial.retry_after.to_s if denial.retry_after.to_i.positive?
|
|
29
|
+
h
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
def self.response(denial) = [denial.status, headers(denial), [denial.body]]
|
|
33
|
+
|
|
34
|
+
# The request as the limiter sees it, from a Rack env.
|
|
35
|
+
def self.request(env, path, remote_ip)
|
|
36
|
+
query = nil
|
|
37
|
+
Request.new(
|
|
38
|
+
env["REQUEST_METHOD"].to_s, path, remote_ip,
|
|
39
|
+
->(name) { header(env, name) },
|
|
40
|
+
lambda do |name|
|
|
41
|
+
query ||= parse_query(env["QUERY_STRING"].to_s)
|
|
42
|
+
query[name].to_s
|
|
43
|
+
end
|
|
44
|
+
)
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
def self.header(env, name)
|
|
48
|
+
key = name.upcase.tr("-", "_")
|
|
49
|
+
key = "HTTP_#{key}" unless key == "CONTENT_TYPE" || key == "CONTENT_LENGTH"
|
|
50
|
+
v = env[key]
|
|
51
|
+
v.is_a?(Array) ? v.first.to_s : v.to_s
|
|
52
|
+
end
|
|
53
|
+
|
|
54
|
+
# First value of each URL-decoded parameter.
|
|
55
|
+
def self.parse_query(qs)
|
|
56
|
+
out = {}
|
|
57
|
+
qs.split("&").each do |pair|
|
|
58
|
+
next if pair.empty?
|
|
59
|
+
k, v = pair.split("=", 2)
|
|
60
|
+
k = decode(k)
|
|
61
|
+
out[k] = decode(v.to_s) unless out.key?(k)
|
|
62
|
+
end
|
|
63
|
+
out
|
|
64
|
+
end
|
|
65
|
+
|
|
66
|
+
def self.decode(s)
|
|
67
|
+
URI.decode_www_form_component(s)
|
|
68
|
+
rescue ArgumentError
|
|
69
|
+
s
|
|
70
|
+
end
|
|
71
|
+
|
|
72
|
+
# One request's rate-limit check, from the middleware to check_rate_limit!.
|
|
73
|
+
class Check
|
|
74
|
+
KEY = :visionary_rate_limit
|
|
75
|
+
|
|
76
|
+
attr_reader :state
|
|
77
|
+
|
|
78
|
+
# Runs the request phase; nil when no rule is in force.
|
|
79
|
+
def self.start(limiter, env, path, remote_ip)
|
|
80
|
+
return nil unless limiter&.active?
|
|
81
|
+
req = RateLimit.request(env, path, remote_ip)
|
|
82
|
+
new(limiter, req, limiter.check_request(req, Time.now.to_f))
|
|
83
|
+
rescue StandardError
|
|
84
|
+
nil
|
|
85
|
+
end
|
|
86
|
+
|
|
87
|
+
def initialize(limiter, req, state)
|
|
88
|
+
@limiter = limiter
|
|
89
|
+
@req = req
|
|
90
|
+
@state = state
|
|
91
|
+
@mutex = Mutex.new
|
|
92
|
+
@user = User.new("", "")
|
|
93
|
+
@done = false
|
|
94
|
+
@raised = nil
|
|
95
|
+
end
|
|
96
|
+
|
|
97
|
+
def denial = @state.denial
|
|
98
|
+
|
|
99
|
+
def set_user(id, role)
|
|
100
|
+
@mutex.synchronize { @user = User.new(id.to_s, role.to_s) }
|
|
101
|
+
end
|
|
102
|
+
|
|
103
|
+
# The user phase, once; raises RateLimited when it (or an earlier call) denied.
|
|
104
|
+
def check_user!(attrs)
|
|
105
|
+
err = @mutex.synchronize do
|
|
106
|
+
unless @done
|
|
107
|
+
@done = true
|
|
108
|
+
begin
|
|
109
|
+
attrs = (attrs || {}).to_h { |k, v| [k.to_s, v.to_s] }
|
|
110
|
+
d = @limiter.check_user(@state, @req, @user, attrs, Time.now.to_f)
|
|
111
|
+
@raised = RateLimited.new(d) if d
|
|
112
|
+
rescue StandardError
|
|
113
|
+
@raised = nil
|
|
114
|
+
end
|
|
115
|
+
end
|
|
116
|
+
@raised
|
|
117
|
+
end
|
|
118
|
+
raise err if err
|
|
119
|
+
end
|
|
120
|
+
|
|
121
|
+
if Fiber.respond_to?(:[])
|
|
122
|
+
def self.current = Fiber[KEY]
|
|
123
|
+
def self.current=(check)
|
|
124
|
+
Fiber[KEY] = check
|
|
125
|
+
end
|
|
126
|
+
else
|
|
127
|
+
def self.current = Thread.current[KEY]
|
|
128
|
+
def self.current=(check)
|
|
129
|
+
Thread.current[KEY] = check
|
|
130
|
+
end
|
|
131
|
+
end
|
|
132
|
+
end
|
|
133
|
+
end
|
|
134
|
+
end
|
data/lib/visionary/version.rb
CHANGED
data/lib/visionary.rb
CHANGED
|
@@ -14,6 +14,7 @@ require_relative "visionary/logs"
|
|
|
14
14
|
require_relative "visionary/retention"
|
|
15
15
|
require_relative "visionary/metrics"
|
|
16
16
|
require_relative "visionary/skip_list"
|
|
17
|
+
require_relative "visionary/rate_limit"
|
|
17
18
|
require_relative "visionary/system/sources"
|
|
18
19
|
require_relative "visionary/system/host_stats"
|
|
19
20
|
require_relative "visionary/system/cgroup"
|
|
@@ -126,6 +127,21 @@ module Visionary
|
|
|
126
127
|
# request seen by Visionary::Middleware.
|
|
127
128
|
def set_user(id, handle: nil, role: nil)
|
|
128
129
|
RequestContext.current&.set_user(id, handle, role)
|
|
130
|
+
RateLimit::Check.current&.set_user(id, role)
|
|
131
|
+
nil
|
|
132
|
+
end
|
|
133
|
+
|
|
134
|
+
# Checks the rate-limit rules that need the user (by user, role or an
|
|
135
|
+
# attribute such as the plan) once set_user has run, and raises
|
|
136
|
+
# Visionary::RateLimited when one limits or blocks the request; the
|
|
137
|
+
# middleware turns it into the response. Only the first call per request
|
|
138
|
+
# counts (later ones repeat its outcome). No-op outside a request seen by
|
|
139
|
+
# Visionary::Middleware.
|
|
140
|
+
#
|
|
141
|
+
# Visionary.set_user(user.id, role: user.role)
|
|
142
|
+
# Visionary.check_rate_limit!(plan: user.plan)
|
|
143
|
+
def check_rate_limit!(attrs = {})
|
|
144
|
+
RateLimit::Check.current&.check_user!(attrs)
|
|
129
145
|
nil
|
|
130
146
|
end
|
|
131
147
|
|
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: visionary-sdk
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.
|
|
4
|
+
version: 1.4.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- styropyr0
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2026-11-
|
|
11
|
+
date: 2026-11-18 00:00:00.000000000 Z
|
|
12
12
|
dependencies: []
|
|
13
13
|
description: |
|
|
14
14
|
The Ruby collector for Visionary. A Rack middleware (with a Railtie for Rails)
|
|
@@ -37,6 +37,10 @@ files:
|
|
|
37
37
|
- lib/visionary/profiler.rb
|
|
38
38
|
- lib/visionary/queue.rb
|
|
39
39
|
- lib/visionary/railtie.rb
|
|
40
|
+
- lib/visionary/rate_limit.rb
|
|
41
|
+
- lib/visionary/rate_limit/config.rb
|
|
42
|
+
- lib/visionary/rate_limit/limiter.rb
|
|
43
|
+
- lib/visionary/rate_limit/poller.rb
|
|
40
44
|
- lib/visionary/redactor.rb
|
|
41
45
|
- lib/visionary/retention.rb
|
|
42
46
|
- lib/visionary/schema.rb
|