gitlab-labkit 5.11.0 → 5.12.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/gitlab-labkit.gemspec +1 -0
- data/lib/labkit/rate_limit/README.md +26 -2
- data/lib/labkit/rate_limit/config/document.rb +64 -0
- data/lib/labkit/rate_limit/config/loader.rb +246 -0
- data/lib/labkit/rate_limit/config/schema.rb +3 -3
- data/lib/labkit/rate_limit/config.rb +2 -0
- data/lib/labkit/rate_limit/evaluator.rb +21 -1
- data/lib/labkit/rate_limit/matcher.rb +44 -10
- data/lib/labkit/rate_limit/rule.rb +28 -12
- metadata +23 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 5f286c7029701485334c9e4f78b26e799a0d74a016e64d7fd7bb881dfb82a853
|
|
4
|
+
data.tar.gz: 8ee6361b4da8dda630297bed4526623f6b53ca813feb2ce2ebc2a5ea4522b1cd
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: cd0a8fd90cee39271ac936e277ff72dde660db5b9d6615860142572a53fd2a7fbda90019b9fad4e0dae62c2c9d20656ae7ef905d588e9bea481b7f206bf890bd
|
|
7
|
+
data.tar.gz: 79e35c32ea2e8e01e0b3df319fe563cbbe43986e2dc2446dd02220c60545d75b8fc2c4a393b1a9eea852b18af8bfb800b3aba3e641c43b2e401cf649cb1d7fea
|
data/gitlab-labkit.gemspec
CHANGED
|
@@ -36,6 +36,7 @@ Gem::Specification.new do |spec|
|
|
|
36
36
|
spec.add_runtime_dependency "opentracing", ">= 0.4", "< 1"
|
|
37
37
|
spec.add_runtime_dependency "pg_query", ">= 6.1.0", "< 7.0"
|
|
38
38
|
spec.add_runtime_dependency "prometheus-client-mmap", ">= 1.2", "< 2.0"
|
|
39
|
+
spec.add_runtime_dependency "re2", ">= 2.15", "< 3"
|
|
39
40
|
spec.add_runtime_dependency "redis", "> 3.0.0", "< 6.0.0"
|
|
40
41
|
spec.add_runtime_dependency "thrift", "< 0.25" # Pinned for jaeger-client 1.1.0 compatibility; thrift 0.25 requires transports to implement message_boundaries?
|
|
41
42
|
|
|
@@ -724,7 +724,7 @@ A gem schema that lacks or changes a vendored field raises `Labkit::RateLimit::C
|
|
|
724
724
|
So upgrade the `gitlab-ratelimit-config-v1` gem in the application first, then the `gitlab-labkit` release that updates the vendored copy.
|
|
725
725
|
|
|
726
726
|
Loading runs in four steps, `parse`, `schema`, `contract` and `rule`, and a failure raises `Labkit::RateLimit::Config::Error` whose `#step` names the step.
|
|
727
|
-
|
|
727
|
+
All four steps exist; the registry that applies a document to a `Limiter` follows in [Part 3 of the plan](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/30263).
|
|
728
728
|
|
|
729
729
|
```ruby
|
|
730
730
|
doc = Labkit::RateLimit::Config::Parser.parse(
|
|
@@ -744,7 +744,31 @@ Inside one message the order is protovalidate's: message rules, then the oneof,
|
|
|
744
744
|
A protobuf map keeps no document order and Go iterates it in an unspecified order, so Ruby reports map entries by sorted key and a cross-SDK comparison matches violations per entry, not by position.
|
|
745
745
|
Map keys in a path are quoted as Go's `strconv.Quote` prints them.
|
|
746
746
|
The check is the strict reading of the schema: a matcher whose operator this binary does not know fails `required`.
|
|
747
|
-
Lenient loading, through the
|
|
747
|
+
Lenient loading, through the `Loader` below, drops such a rule with a warning before the schema step, so a newer operator does not reject the file.
|
|
748
|
+
`Labkit::RateLimit::Config::Loader` runs all four steps and returns a `Document`:
|
|
749
|
+
|
|
750
|
+
```ruby
|
|
751
|
+
loader = Labkit::RateLimit::Config::Loader.new(strict: Rails.env.test?, logger: Gitlab::AppLogger)
|
|
752
|
+
doc = loader.load(contracts: Dir["config/rate_limits/*.yaml"], rate_limits: "config/rate_limits.yaml")
|
|
753
|
+
doc.limiter?("rack_request") # => true
|
|
754
|
+
doc.available_properties("rack_request") # => [:plan, :path, :requester_id]
|
|
755
|
+
doc.default_rules("rack_request") # => [Rule, ...] from the contract, document order
|
|
756
|
+
doc.override_rules("rack_request") # => [Rule, ...] from the rate limits file
|
|
757
|
+
doc.warnings # => [{ step:, limiter:, rule:, message: }, ...]
|
|
758
|
+
```
|
|
759
|
+
|
|
760
|
+
The contract step merges the contract files: a limiter is declared, with its description, in exactly one file, a later file may add properties and append default rules, and every rule may use only the properties its limiter lists.
|
|
761
|
+
It rejects an override that takes a protected name or carries a `position`, and warns when an override drops a match key of the default it replaces.
|
|
762
|
+
A contract mistake rejects in both modes, because the contract is the application's own.
|
|
763
|
+
In lenient mode a rule for an unknown limiter, with an unknown property, with a protected name, or with a matcher whose only operator this version does not know is dropped with a warning, and a position is ignored; any other failure rejects the whole rate limits file.
|
|
764
|
+
Every warning is also logged as `rate_limit_config_warning` with `step`, `limiter`, `rule` and `detail`.
|
|
765
|
+
|
|
766
|
+
The rule step builds each proto rule as a `Rule` with `source: :file`.
|
|
767
|
+
A file rule compares identifier values as strings, as every SDK does for a rule from a document: `{ eq: "false" }` matches a `false` fact, a missing key reads as `""`, an integer reads as its decimal digits and a symbol as its name; other value types fall back to `to_s` and have no fixed cross-SDK form, so a host keeps them out of identifiers that file rules match on.
|
|
768
|
+
A rule built in Ruby keeps Ruby equality.
|
|
769
|
+
A file rule checks its name in every environment and compiles `re` and `nre` with the `re2` gem, so a pattern that only Ruby accepts, such as a lookbehind, is rejected at step `rule`, and `^` anchors the whole text as it does in Go and Rust.
|
|
770
|
+
`Document.empty` is a document with no limiters, for a host that has no contract yet.
|
|
771
|
+
|
|
748
772
|
`scripts/vendor-ratelimit-proto.sh` also copies `config.proto` and `constraint.proto` next to the generated code, and `schema_spec.rb` parses them and compares every field rule (caps, bounds, presets, flags), every message rule id and message, and the name pattern with `Schema`, so a `PROTOS_REF` bump that changes any of those fails the suite until `Schema` follows.
|
|
749
773
|
The CEL expressions themselves are mirrored by hand and pinned by the examples.
|
|
750
774
|
|
|
@@ -0,0 +1,64 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Labkit
|
|
4
|
+
module RateLimit
|
|
5
|
+
module Config
|
|
6
|
+
# Document is what Loader#load returns: the merged contract and the rules
|
|
7
|
+
# of the rate limits file, per limiter, as Rule objects a Limiter can use.
|
|
8
|
+
# It is immutable; a host builds a new one to reload.
|
|
9
|
+
class Document
|
|
10
|
+
# One limiter's slice of the document.
|
|
11
|
+
Limiter = Data.define(:properties, :default_rules, :override_rules, :protected_rules)
|
|
12
|
+
|
|
13
|
+
attr_reader :warnings
|
|
14
|
+
|
|
15
|
+
# A document with no limiters: every host default applies unchanged.
|
|
16
|
+
def self.empty
|
|
17
|
+
new({}, [])
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
# @param limiters [Hash{String => Limiter}]
|
|
21
|
+
# @param warnings [Array<Hash>] with :step, :limiter, :rule, :message
|
|
22
|
+
def initialize(limiters, warnings)
|
|
23
|
+
@limiters = limiters.freeze
|
|
24
|
+
@warnings = warnings.freeze
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
def limiter?(name)
|
|
28
|
+
@limiters.key?(name.to_s)
|
|
29
|
+
end
|
|
30
|
+
|
|
31
|
+
# @return [Array<String>] every limiter the contract declares
|
|
32
|
+
def limiters
|
|
33
|
+
@limiters.keys
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
# @return [Array<Symbol>] the properties a rule of this limiter may use
|
|
37
|
+
def available_properties(name)
|
|
38
|
+
slice(name)&.properties || []
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
# @return [Array<Rule>] the file rules for this limiter, document order
|
|
42
|
+
def override_rules(name)
|
|
43
|
+
slice(name)&.override_rules || []
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
# @return [Array<Rule>] the contract's default rules, document order
|
|
47
|
+
def default_rules(name)
|
|
48
|
+
slice(name)&.default_rules || []
|
|
49
|
+
end
|
|
50
|
+
|
|
51
|
+
# @return [Array<String>] default rule names no override may replace
|
|
52
|
+
def protected_rules(name)
|
|
53
|
+
slice(name)&.protected_rules || []
|
|
54
|
+
end
|
|
55
|
+
|
|
56
|
+
private
|
|
57
|
+
|
|
58
|
+
def slice(name)
|
|
59
|
+
@limiters[name.to_s]
|
|
60
|
+
end
|
|
61
|
+
end
|
|
62
|
+
end
|
|
63
|
+
end
|
|
64
|
+
end
|
|
@@ -0,0 +1,246 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Labkit
|
|
4
|
+
module RateLimit
|
|
5
|
+
module Config
|
|
6
|
+
# Loader runs the four steps of the config loader over a set of contract
|
|
7
|
+
# files and an optional rate limits file, and returns a Document.
|
|
8
|
+
#
|
|
9
|
+
# parse and schema are Parser and Schema. The contract step lives here: it
|
|
10
|
+
# merges the contract files, checks that every rule uses only properties
|
|
11
|
+
# its limiter lists, that no override takes a protected name, and that a
|
|
12
|
+
# file rule carries no position. The rule step converts each proto Rule
|
|
13
|
+
# into a Rule with source: :file, which compiles patterns with RE2.
|
|
14
|
+
#
|
|
15
|
+
# Strict mode, for CI, rejects on any failure. Lenient mode, for a running
|
|
16
|
+
# process, drops with a warning a rule for an unknown limiter, with an
|
|
17
|
+
# unknown property, with a protected name, or with a matcher whose only
|
|
18
|
+
# operator this binary does not know, and ignores a position. Any other
|
|
19
|
+
# failure rejects the whole file: the host then logs and runs on its
|
|
20
|
+
# defaults. Contract files are the application's own, so a contract
|
|
21
|
+
# mistake rejects in both modes.
|
|
22
|
+
class Loader
|
|
23
|
+
ACTIONS = { ACTION_LIMIT: :limit, ACTION_LOG: :log, ACTION_SKIP: :skip }.freeze
|
|
24
|
+
# The operators this version converts. A newer protos gem can decode
|
|
25
|
+
# an operator or an action outside these sets.
|
|
26
|
+
OPERATORS = %i[eq ne re nre one_of none_of all].freeze
|
|
27
|
+
|
|
28
|
+
# Merged view of one limiter across the contract files.
|
|
29
|
+
Spec = Struct.new(:properties, :default_rules, :protected_rules, :described)
|
|
30
|
+
|
|
31
|
+
# Default rules and properties per limiter, across every contract
|
|
32
|
+
# file. Schema caps one file's list; the merge must hold the same line.
|
|
33
|
+
MAX_DEFAULT_RULES = Schema::FIELD_RULES.fetch("LimiterSpec.default_rules").fetch(:max_items)
|
|
34
|
+
MAX_PROPERTIES = Schema::FIELD_RULES.fetch("LimiterSpec.available_properties").fetch(:max_items)
|
|
35
|
+
|
|
36
|
+
# @param strict [Boolean] reject on any failure instead of dropping rules
|
|
37
|
+
# @param logger [#warn, nil] receives each warning as it is recorded
|
|
38
|
+
def initialize(strict: false, logger: nil)
|
|
39
|
+
@strict = strict
|
|
40
|
+
@logger = logger || RateLimit.config.logger || Labkit::Logging::JsonLogger.new($stdout)
|
|
41
|
+
# load keeps the run's warnings on the instance, so two threads
|
|
42
|
+
# sharing one Loader would mix them.
|
|
43
|
+
@mutex = Mutex.new
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
# @param contracts [Array<String>] paths of available_limiters files, in order
|
|
47
|
+
# @param rate_limits [String, nil] path of the rate_limits file
|
|
48
|
+
# @return [Document]
|
|
49
|
+
# @raise [Error] with the step that failed
|
|
50
|
+
def load(contracts:, rate_limits: nil)
|
|
51
|
+
@mutex.synchronize { load_locked(contracts, rate_limits) }
|
|
52
|
+
end
|
|
53
|
+
|
|
54
|
+
private
|
|
55
|
+
|
|
56
|
+
attr_reader :strict
|
|
57
|
+
|
|
58
|
+
def load_locked(contracts, rate_limits)
|
|
59
|
+
@warnings = []
|
|
60
|
+
specs = merge_contracts(contracts)
|
|
61
|
+
overrides = rate_limits ? load_rate_limits(rate_limits, specs) : {}
|
|
62
|
+
|
|
63
|
+
limiters = specs.to_h do |name, spec|
|
|
64
|
+
defaults = spec.default_rules.map { |rule| convert(name, rule) }
|
|
65
|
+
[name, Document::Limiter.new(spec.properties.freeze, defaults.freeze, overrides.fetch(name, []).freeze, spec.protected_rules.freeze)]
|
|
66
|
+
end
|
|
67
|
+
Document.new(limiters, @warnings)
|
|
68
|
+
end
|
|
69
|
+
|
|
70
|
+
def merge_contracts(paths)
|
|
71
|
+
specs = {}
|
|
72
|
+
paths.each do |path|
|
|
73
|
+
doc = Schema.check!(Parser.parse(path, Proto::AvailableLimiters, strict: true))
|
|
74
|
+
doc.available_limiters.each { |name, proto_spec| merge_limiter(specs, name, proto_spec, path) }
|
|
75
|
+
end
|
|
76
|
+
specs.each do |name, spec|
|
|
77
|
+
contract_error("limiter #{name} has no description in any contract file") unless spec.described
|
|
78
|
+
check_caps!(name, spec)
|
|
79
|
+
spec.default_rules.each { |rule| check_properties!(name, rule, spec) }
|
|
80
|
+
end
|
|
81
|
+
specs
|
|
82
|
+
end
|
|
83
|
+
|
|
84
|
+
# A limiter is declared, with its description, in exactly one file. A
|
|
85
|
+
# later file may add properties and append default rules.
|
|
86
|
+
def merge_limiter(specs, name, proto_spec, path)
|
|
87
|
+
spec = specs[name] ||= Spec.new([], [], [], false)
|
|
88
|
+
unless proto_spec.description.empty?
|
|
89
|
+
contract_error("limiter #{name} is declared twice, again in #{path}") if spec.described
|
|
90
|
+
spec.described = true
|
|
91
|
+
end
|
|
92
|
+
|
|
93
|
+
proto_spec.available_properties.each { |p| spec.properties << p.to_sym unless spec.properties.include?(p.to_sym) }
|
|
94
|
+
proto_spec.default_rules.each do |rule|
|
|
95
|
+
contract_error("limiter #{name}: default rule #{rule.name} is defined twice, again in #{path}") if spec.default_rules.any? { |r| r.name == rule.name }
|
|
96
|
+
spec.default_rules << rule
|
|
97
|
+
end
|
|
98
|
+
spec.protected_rules.concat(proto_spec.protected_rules.to_a - spec.protected_rules)
|
|
99
|
+
end
|
|
100
|
+
|
|
101
|
+
def load_rate_limits(path, specs)
|
|
102
|
+
doc = Parser.parse(path, Proto::RateLimits, strict: strict)
|
|
103
|
+
drop_unknown_operators(doc) unless strict
|
|
104
|
+
Schema.check!(doc)
|
|
105
|
+
|
|
106
|
+
# Map#to_h ignores a block, so the overrides are collected by hand.
|
|
107
|
+
overrides = {}
|
|
108
|
+
doc.rate_limits.each do |name, list|
|
|
109
|
+
spec = specs[name]
|
|
110
|
+
if spec
|
|
111
|
+
overrides[name] = list.rules.filter_map { |rule| override_for(name, rule, spec) }
|
|
112
|
+
else
|
|
113
|
+
contract_error("rate_limits names limiter #{name}, which no contract declares") if strict
|
|
114
|
+
warn(:contract, name, "", "limiter #{name} is not in the contract; its rules are dropped")
|
|
115
|
+
end
|
|
116
|
+
end
|
|
117
|
+
overrides
|
|
118
|
+
end
|
|
119
|
+
|
|
120
|
+
# An unknown operator is an empty oneof after the parse step, which the
|
|
121
|
+
# schema step would reject as a missing operator for the whole file.
|
|
122
|
+
def drop_unknown_operators(doc)
|
|
123
|
+
doc.rate_limits.each do |name, list|
|
|
124
|
+
kept = list.rules.reject do |rule|
|
|
125
|
+
dropped = rule.match.any? { |_, matcher| unknown_operator?(matcher) }
|
|
126
|
+
warn(:parse, name, rule.name, "a matcher uses an operator this version does not know; rule dropped") if dropped
|
|
127
|
+
dropped
|
|
128
|
+
end
|
|
129
|
+
list.rules.replace(kept)
|
|
130
|
+
end
|
|
131
|
+
end
|
|
132
|
+
|
|
133
|
+
def unknown_operator?(matcher)
|
|
134
|
+
return true unless OPERATORS.include?(matcher.op)
|
|
135
|
+
|
|
136
|
+
matcher.op == :all && matcher.all.matchers.any? { |m| unknown_operator?(m) }
|
|
137
|
+
end
|
|
138
|
+
|
|
139
|
+
# The contract checks on one file rule. Returns the Rule, or nil when
|
|
140
|
+
# lenient mode drops it.
|
|
141
|
+
def override_for(name, rule, spec)
|
|
142
|
+
return reject(:contract, name, rule.name, "#{rule.name} is a protected rule of #{name}; the contract alone sets its limit") if spec.protected_rules.include?(rule.name)
|
|
143
|
+
|
|
144
|
+
unknown = unknown_property(rule, spec)
|
|
145
|
+
return reject(:contract, name, rule.name, "property #{unknown} is not listed for limiter #{name}") if unknown
|
|
146
|
+
|
|
147
|
+
if rule.has_position?
|
|
148
|
+
contract_error("limiter #{name}: rule #{rule.name} has a position; only a published rule can") if strict
|
|
149
|
+
warn(:contract, name, rule.name, "position is ignored on a file rule")
|
|
150
|
+
end
|
|
151
|
+
|
|
152
|
+
warn_dropped_match_keys(name, rule, spec)
|
|
153
|
+
convert(name, rule)
|
|
154
|
+
end
|
|
155
|
+
|
|
156
|
+
# An override that matches less than the default it replaces widens the
|
|
157
|
+
# rule. Allowed, since every matching rule counts, but worth a line in
|
|
158
|
+
# the log.
|
|
159
|
+
def warn_dropped_match_keys(name, rule, spec)
|
|
160
|
+
default = spec.default_rules.find { |d| d.name == rule.name }
|
|
161
|
+
return unless default
|
|
162
|
+
|
|
163
|
+
dropped = default.match.keys - rule.match.keys
|
|
164
|
+
warn(:contract, name, rule.name, "dropped match keys: #{dropped.sort.join(', ')}") unless dropped.empty?
|
|
165
|
+
end
|
|
166
|
+
|
|
167
|
+
def check_caps!(name, spec)
|
|
168
|
+
rules = spec.default_rules.size
|
|
169
|
+
contract_error("limiter #{name} has #{rules} default rules across the contract files; at most #{MAX_DEFAULT_RULES} are allowed") if rules > MAX_DEFAULT_RULES
|
|
170
|
+
|
|
171
|
+
properties = spec.properties.size
|
|
172
|
+
contract_error("limiter #{name} lists #{properties} properties across the contract files; at most #{MAX_PROPERTIES} are allowed") if properties > MAX_PROPERTIES
|
|
173
|
+
end
|
|
174
|
+
|
|
175
|
+
def check_properties!(name, rule, spec)
|
|
176
|
+
unknown = unknown_property(rule, spec)
|
|
177
|
+
contract_error("limiter #{name}: default rule #{rule.name} uses property #{unknown}, which the limiter does not list") if unknown
|
|
178
|
+
end
|
|
179
|
+
|
|
180
|
+
def unknown_property(rule, spec)
|
|
181
|
+
(used_properties(rule) - spec.properties).first
|
|
182
|
+
end
|
|
183
|
+
|
|
184
|
+
def used_properties(rule)
|
|
185
|
+
used = rule.characteristics.map(&:to_sym) + rule.match.keys.map(&:to_sym)
|
|
186
|
+
used << rule.count_distinct.to_sym unless rule.count_distinct.empty?
|
|
187
|
+
used
|
|
188
|
+
end
|
|
189
|
+
|
|
190
|
+
def convert(limiter, rule)
|
|
191
|
+
Rule.new(
|
|
192
|
+
name: rule.name,
|
|
193
|
+
action: ACTIONS.fetch(rule.action) { raise ArgumentError, "unknown action #{rule.action.inspect}" },
|
|
194
|
+
limit: rule.has_limit? ? rule.limit : nil,
|
|
195
|
+
period: rule.has_period_s? ? rule.period_s : nil,
|
|
196
|
+
ban_for: rule.has_ban_for_s? ? rule.ban_for_s : nil,
|
|
197
|
+
characteristics: rule.characteristics.to_a,
|
|
198
|
+
count_distinct: rule.count_distinct.empty? ? nil : rule.count_distinct,
|
|
199
|
+
match: match_input(rule.match),
|
|
200
|
+
source: :file
|
|
201
|
+
)
|
|
202
|
+
rescue ArgumentError => e
|
|
203
|
+
raise Error.new(:rule, "#{limiter}/#{rule.name}: #{e.message}")
|
|
204
|
+
end
|
|
205
|
+
|
|
206
|
+
# Map#to_h would turn the nested messages into plain Hashes, so the
|
|
207
|
+
# entries are walked as messages.
|
|
208
|
+
def match_input(map)
|
|
209
|
+
input = {}
|
|
210
|
+
map.each { |key, matcher| input[key] = matcher_input(matcher) }
|
|
211
|
+
input
|
|
212
|
+
end
|
|
213
|
+
|
|
214
|
+
# The proto Matcher in the shape Rule.new accepts.
|
|
215
|
+
def matcher_input(matcher)
|
|
216
|
+
case matcher.op
|
|
217
|
+
when :eq, :ne, :re, :nre then { matcher.op => matcher[matcher.op.to_s] }
|
|
218
|
+
when :one_of then { oneOf: matcher.one_of.values.to_a }
|
|
219
|
+
when :none_of then { noneOf: matcher.none_of.values.to_a }
|
|
220
|
+
when :all then matcher.all.matchers.map { |m| matcher_input(m) }
|
|
221
|
+
else raise ArgumentError, "unknown matcher operator #{matcher.op.inspect}"
|
|
222
|
+
end
|
|
223
|
+
end
|
|
224
|
+
|
|
225
|
+
def reject(step, limiter, rule, message)
|
|
226
|
+
contract_error("limiter #{limiter}: rule #{rule}: #{message}") if strict
|
|
227
|
+
|
|
228
|
+
warn(step, limiter, rule, message)
|
|
229
|
+
nil
|
|
230
|
+
end
|
|
231
|
+
|
|
232
|
+
def warn(step, limiter, rule, message)
|
|
233
|
+
warning = { step: step, limiter: limiter, rule: rule, message: message }
|
|
234
|
+
@warnings << warning
|
|
235
|
+
# The warning's own :message is the detail; the log line keeps the
|
|
236
|
+
# fixed message every rate_limit log line has.
|
|
237
|
+
@logger.warn(message: "rate_limit_config_warning", step: step, limiter: limiter, rule: rule, detail: message)
|
|
238
|
+
end
|
|
239
|
+
|
|
240
|
+
def contract_error(message)
|
|
241
|
+
raise Error.new(:contract, message)
|
|
242
|
+
end
|
|
243
|
+
end
|
|
244
|
+
end
|
|
245
|
+
end
|
|
246
|
+
end
|
|
@@ -19,9 +19,9 @@ module Labkit
|
|
|
19
19
|
# cap, a flag, a rule id or a message fails the suite until this file
|
|
20
20
|
# follows. The CEL expressions have no such check; the examples pin them.
|
|
21
21
|
#
|
|
22
|
-
# The check is the strict reading of the schema. Lenient loading
|
|
23
|
-
# rule whose matcher has no known operator before this step
|
|
24
|
-
#
|
|
22
|
+
# The check is the strict reading of the schema. Lenient loading (Loader)
|
|
23
|
+
# drops a rule whose matcher has no known operator before this step,
|
|
24
|
+
# because such a matcher fails `required` here.
|
|
25
25
|
module Schema
|
|
26
26
|
# for_key marks a violation on a map key; its path is the entry's path.
|
|
27
27
|
Violation = Data.define(:path, :rule_id, :message, :for_key) do
|
|
@@ -16,6 +16,8 @@ module Labkit
|
|
|
16
16
|
Proto = Gitlab::Ratelimit::Config::V1
|
|
17
17
|
|
|
18
18
|
autoload :Error, "labkit/rate_limit/config/error"
|
|
19
|
+
autoload :Document, "labkit/rate_limit/config/document"
|
|
20
|
+
autoload :Loader, "labkit/rate_limit/config/loader"
|
|
19
21
|
autoload :Parser, "labkit/rate_limit/config/parser"
|
|
20
22
|
autoload :Schema, "labkit/rate_limit/config/schema"
|
|
21
23
|
end
|
|
@@ -321,8 +321,28 @@ module Labkit
|
|
|
321
321
|
cursor.rule = nil unless e
|
|
322
322
|
end
|
|
323
323
|
|
|
324
|
+
# A file rule compares as strings, the way every SDK does for a rule
|
|
325
|
+
# from a document. A code rule keeps Ruby equality, so {eq: true}
|
|
326
|
+
# matches true only.
|
|
324
327
|
def rule_matches?(rule, identifier)
|
|
325
|
-
rule.
|
|
328
|
+
if rule.source == :file
|
|
329
|
+
rule.match.all? { |key, matcher| matcher.match?(file_value(identifier[key])) }
|
|
330
|
+
else
|
|
331
|
+
rule.match.all? { |key, matcher| matcher.match?(identifier[key]) }
|
|
332
|
+
end
|
|
333
|
+
end
|
|
334
|
+
|
|
335
|
+
# The string form of an identifier value for a file rule. A missing key
|
|
336
|
+
# reads as "". Strings, symbols, booleans and integers have one form in
|
|
337
|
+
# every SDK; anything else falls back to to_s and is the host's to keep
|
|
338
|
+
# out of identifiers.
|
|
339
|
+
def file_value(value)
|
|
340
|
+
case value
|
|
341
|
+
when nil then ""
|
|
342
|
+
when String then value
|
|
343
|
+
when Symbol then value.name
|
|
344
|
+
else value.to_s
|
|
345
|
+
end
|
|
326
346
|
end
|
|
327
347
|
|
|
328
348
|
def missing_count_distinct_value?(rule, identifier)
|
|
@@ -47,6 +47,12 @@ module Labkit
|
|
|
47
47
|
# presence matcher alongside the exclusions:
|
|
48
48
|
#
|
|
49
49
|
# path: [/./, { nre: '\A/api/' }]
|
|
50
|
+
#
|
|
51
|
+
# Rules from a config file are built with engine: :re2. Their :re and :nre
|
|
52
|
+
# patterns compile with RE2, the syntax every SDK shares, so a pattern that
|
|
53
|
+
# only Ruby accepts (lookbehind, a possessive quantifier) is rejected and a
|
|
54
|
+
# `^` anchors the whole text, not each line. RE2 has no backtracking, so
|
|
55
|
+
# those matchers need no timeout.
|
|
50
56
|
class Matcher < Data.define(:type, :value)
|
|
51
57
|
KNOWN_HASH_KEYS = %i[eq ne re nre oneOf noneOf].freeze
|
|
52
58
|
ERROR_INSPECT_LIMIT = 80
|
|
@@ -74,22 +80,27 @@ module Labkit
|
|
|
74
80
|
# nothing in practice, since callers pass Regexp constants.
|
|
75
81
|
MATCH_TIMEOUT_SECONDS = 0.05
|
|
76
82
|
|
|
77
|
-
|
|
83
|
+
ENGINES = %i[ruby re2].freeze
|
|
84
|
+
|
|
85
|
+
# engine is positional, not a keyword: callers write build(eq: x) and a
|
|
86
|
+
# keyword here would swallow that Hash.
|
|
87
|
+
def self.build(input, engine = :ruby)
|
|
88
|
+
raise ArgumentError, "engine must be one of #{ENGINES.inspect}, got #{engine.inspect}" unless ENGINES.include?(engine)
|
|
78
89
|
raise ArgumentError, nesting_error(input) if too_deeply_nested?(input)
|
|
79
90
|
|
|
80
91
|
case input
|
|
81
92
|
when Regexp
|
|
82
|
-
new(type: :re, value:
|
|
93
|
+
new(type: :re, value: regexp(:re, input, engine))
|
|
83
94
|
when Hash
|
|
84
|
-
from_hash(input)
|
|
95
|
+
from_hash(input, engine)
|
|
85
96
|
when Array
|
|
86
|
-
compose(input)
|
|
97
|
+
compose(input, engine)
|
|
87
98
|
else
|
|
88
99
|
new(type: :eq, value: input)
|
|
89
100
|
end
|
|
90
101
|
end
|
|
91
102
|
|
|
92
|
-
def self.from_hash(input)
|
|
103
|
+
def self.from_hash(input, engine)
|
|
93
104
|
raise ArgumentError, invalid_shape_error(input) if input.size != 1
|
|
94
105
|
|
|
95
106
|
type, source = input.first
|
|
@@ -109,11 +120,11 @@ module Labkit
|
|
|
109
120
|
"rate-limit match value has unknown type key #{truncate_for_error(type)}; accepted: #{KNOWN_HASH_KEYS.inspect}"
|
|
110
121
|
end
|
|
111
122
|
|
|
112
|
-
compile(type_sym, source)
|
|
123
|
+
compile(type_sym, source, engine)
|
|
113
124
|
end
|
|
114
125
|
private_class_method :from_hash
|
|
115
126
|
|
|
116
|
-
def self.compile(type_sym, source)
|
|
127
|
+
def self.compile(type_sym, source, engine)
|
|
117
128
|
case type_sym
|
|
118
129
|
when :eq, :ne
|
|
119
130
|
new(type: type_sym, value: source)
|
|
@@ -134,14 +145,19 @@ module Labkit
|
|
|
134
145
|
|
|
135
146
|
new(type: type_sym, value: Set.new(source).freeze)
|
|
136
147
|
when :re, :nre
|
|
137
|
-
new(type: type_sym, value:
|
|
148
|
+
new(type: type_sym, value: regexp(type_sym, source, engine))
|
|
138
149
|
end
|
|
139
150
|
end
|
|
140
151
|
private_class_method :compile
|
|
141
152
|
|
|
153
|
+
def self.regexp(key, source, engine)
|
|
154
|
+
engine == :re2 ? re2_regexp(key, source) : with_match_timeout(key, source)
|
|
155
|
+
end
|
|
156
|
+
private_class_method :regexp
|
|
157
|
+
|
|
142
158
|
# Builds the :all matcher behind the Array form. Entries go back through
|
|
143
159
|
# .build, which is what keeps a nested regex bounded by the match timeout.
|
|
144
|
-
def self.compose(input)
|
|
160
|
+
def self.compose(input, engine)
|
|
145
161
|
raise ArgumentError, "rate-limit match value [] must not be empty" if input.empty?
|
|
146
162
|
|
|
147
163
|
unless input.all? { |entry| entry.is_a?(Hash) || entry.is_a?(Regexp) }
|
|
@@ -150,10 +166,28 @@ module Labkit
|
|
|
150
166
|
"got #{truncate_for_error(input)}"
|
|
151
167
|
end
|
|
152
168
|
|
|
153
|
-
new(type: :all, value: input.map { |entry| build(entry) }.freeze)
|
|
169
|
+
new(type: :all, value: input.map { |entry| build(entry, engine) }.freeze)
|
|
154
170
|
end
|
|
155
171
|
private_class_method :compose
|
|
156
172
|
|
|
173
|
+
# The re2 gem is loaded here, not at require time, so a host that builds
|
|
174
|
+
# no file rules never pays for it.
|
|
175
|
+
# A Ruby Regexp with options cannot be handed to RE2 as its source alone:
|
|
176
|
+
# /api/i would silently turn case-sensitive.
|
|
177
|
+
def self.re2_regexp(key, source)
|
|
178
|
+
require "re2"
|
|
179
|
+
flagged = source.is_a?(Regexp) && source.options.anybits?(Regexp::IGNORECASE | Regexp::MULTILINE | Regexp::EXTENDED)
|
|
180
|
+
raise ArgumentError, "rate-limit match value {#{key}: #{truncate_for_error(source)}} has Regexp options, which RE2 does not take; write them into the pattern" if flagged
|
|
181
|
+
|
|
182
|
+
pattern = source.is_a?(Regexp) ? source.source : source.to_s
|
|
183
|
+
compiled = RE2::Regexp.new(pattern, log_errors: false)
|
|
184
|
+
return compiled if compiled.ok?
|
|
185
|
+
|
|
186
|
+
raise ArgumentError,
|
|
187
|
+
"rate-limit match value {#{key}: #{truncate_for_error(source)}} failed to compile: #{compiled.error}"
|
|
188
|
+
end
|
|
189
|
+
private_class_method :re2_regexp
|
|
190
|
+
|
|
157
191
|
# Compiles +source+ (a String pattern or an existing Regexp) into a
|
|
158
192
|
# Regexp bounded by MATCH_TIMEOUT_SECONDS.
|
|
159
193
|
#
|
|
@@ -41,11 +41,16 @@ module Labkit
|
|
|
41
41
|
# admitting rules after counter rules. Default false; nil
|
|
42
42
|
# means not set.
|
|
43
43
|
#
|
|
44
|
+
# source - :code for a rule built in Ruby (default), :file for a rule
|
|
45
|
+
# loaded from a rate limit config document. A :file rule
|
|
46
|
+
# compiles its patterns with RE2 and checks its name in
|
|
47
|
+
# every environment, because an operator wrote it.
|
|
48
|
+
#
|
|
44
49
|
# +name+ must be a lowercase alphanumeric-and-underscore string of at most 64
|
|
45
50
|
# characters. It is used as the middle segment of every Redis counter key for
|
|
46
51
|
# this rule, so changing a rule's name mid-window abandons its in-flight counters.
|
|
47
52
|
Rule = Data.define(
|
|
48
|
-
:name, :match, :limit, :period, :action, :characteristics, :count_distinct, :ban_for, :admit_counted_values
|
|
53
|
+
:name, :match, :limit, :period, :action, :characteristics, :count_distinct, :ban_for, :admit_counted_values, :source
|
|
49
54
|
) do
|
|
50
55
|
# ban_for changes the accounting; action still decides who gets blocked.
|
|
51
56
|
# A :skip rule counts nothing, so it has no limit to cross and no ban to
|
|
@@ -106,36 +111,47 @@ module Labkit
|
|
|
106
111
|
sym
|
|
107
112
|
end
|
|
108
113
|
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
)
|
|
114
|
+
# A :file rule is checked in every environment: an operator wrote it, and
|
|
115
|
+
# Limiter's production fallback of sanitising the name would silently
|
|
116
|
+
# change the counter key of a rule the operator can see in the file.
|
|
117
|
+
def self.validate_name!(name, source)
|
|
113
118
|
raise ArgumentError, "name must be a String or Symbol, got #{name.class}" unless name.is_a?(String) || name.is_a?(Symbol)
|
|
114
119
|
|
|
115
120
|
name_str = name.to_s
|
|
116
121
|
raise ArgumentError, "name must not be empty" if name_str.empty?
|
|
122
|
+
return name_str unless source == :file || Labkit.dev_or_test?
|
|
123
|
+
|
|
124
|
+
raise ArgumentError, "Invalid rule name: #{name.inspect}. Must match /\\A[a-z0-9_]+\\z/" unless RULE_NAME_PATTERN.match?(name_str)
|
|
125
|
+
raise ArgumentError, "Rule name too long: #{name.inspect}. Maximum 64 characters" if name_str.length > RULE_NAME_MAX_LENGTH
|
|
126
|
+
|
|
127
|
+
name_str
|
|
128
|
+
end
|
|
129
|
+
|
|
130
|
+
def initialize( # rubocop:disable Metrics/ParameterLists -- one keyword per Rule field; the public API grows by keywords with defaults
|
|
131
|
+
name:, limit:, period:, characteristics:,
|
|
132
|
+
match: {}, action: :limit, count_distinct: nil, ban_for: nil, admit_counted_values: false, source: :code
|
|
133
|
+
)
|
|
134
|
+
raise ArgumentError, "source must be :code or :file, got #{source.inspect}" unless %i[code file].include?(source)
|
|
135
|
+
|
|
136
|
+
name_str = self.class.validate_name!(name, source)
|
|
117
137
|
|
|
118
138
|
action_sym = action.to_sym
|
|
119
139
|
raise ArgumentError, "Invalid action: #{action.inspect}. Must be one of: #{KNOWN_ACTIONS.inspect}" unless KNOWN_ACTIONS.include?(action_sym)
|
|
120
140
|
|
|
121
|
-
if Labkit.dev_or_test?
|
|
122
|
-
raise ArgumentError, "Invalid rule name: #{name.inspect}. Must match /\\A[a-z0-9_]+\\z/" unless RULE_NAME_PATTERN.match?(name_str)
|
|
123
|
-
raise ArgumentError, "Rule name too long: #{name.inspect}. Maximum 64 characters" if name_str.length > RULE_NAME_MAX_LENGTH
|
|
124
|
-
end
|
|
125
|
-
|
|
126
141
|
characteristics_arr = Array(characteristics).map(&:to_sym).freeze
|
|
127
142
|
count_distinct_sym = self.class.normalize_count_distinct(count_distinct, characteristics_arr)
|
|
128
143
|
|
|
129
144
|
super(
|
|
130
145
|
name: name_str.freeze,
|
|
131
|
-
match: match.transform_keys(&:to_sym).transform_values { |v| Matcher.build(v) }.freeze,
|
|
146
|
+
match: match.transform_keys(&:to_sym).transform_values { |v| Matcher.build(v, source == :file ? :re2 : :ruby) }.freeze,
|
|
132
147
|
limit: self.class.normalize_numeric(:limit, limit, action_sym),
|
|
133
148
|
period: self.class.normalize_numeric(:period, period, action_sym),
|
|
134
149
|
action: action_sym,
|
|
135
150
|
characteristics: characteristics_arr,
|
|
136
151
|
count_distinct: count_distinct_sym,
|
|
137
152
|
ban_for: self.class.normalize_ban_for(ban_for, action_sym, count_distinct_sym),
|
|
138
|
-
admit_counted_values: self.class.normalize_admit_counted_values(admit_counted_values, action_sym, count_distinct_sym)
|
|
153
|
+
admit_counted_values: self.class.normalize_admit_counted_values(admit_counted_values, action_sym, count_distinct_sym),
|
|
154
|
+
source: source
|
|
139
155
|
)
|
|
140
156
|
end
|
|
141
157
|
end
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: gitlab-labkit
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 5.
|
|
4
|
+
version: 5.12.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Andrew Newdigate
|
|
@@ -257,6 +257,26 @@ dependencies:
|
|
|
257
257
|
- - "<"
|
|
258
258
|
- !ruby/object:Gem::Version
|
|
259
259
|
version: '2.0'
|
|
260
|
+
- !ruby/object:Gem::Dependency
|
|
261
|
+
name: re2
|
|
262
|
+
requirement: !ruby/object:Gem::Requirement
|
|
263
|
+
requirements:
|
|
264
|
+
- - ">="
|
|
265
|
+
- !ruby/object:Gem::Version
|
|
266
|
+
version: '2.15'
|
|
267
|
+
- - "<"
|
|
268
|
+
- !ruby/object:Gem::Version
|
|
269
|
+
version: '3'
|
|
270
|
+
type: :runtime
|
|
271
|
+
prerelease: false
|
|
272
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
273
|
+
requirements:
|
|
274
|
+
- - ">="
|
|
275
|
+
- !ruby/object:Gem::Version
|
|
276
|
+
version: '2.15'
|
|
277
|
+
- - "<"
|
|
278
|
+
- !ruby/object:Gem::Version
|
|
279
|
+
version: '3'
|
|
260
280
|
- !ruby/object:Gem::Dependency
|
|
261
281
|
name: redis
|
|
262
282
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -652,7 +672,9 @@ files:
|
|
|
652
672
|
- lib/labkit/rate_limit/README.md
|
|
653
673
|
- lib/labkit/rate_limit/block_summary.rb
|
|
654
674
|
- lib/labkit/rate_limit/config.rb
|
|
675
|
+
- lib/labkit/rate_limit/config/document.rb
|
|
655
676
|
- lib/labkit/rate_limit/config/error.rb
|
|
677
|
+
- lib/labkit/rate_limit/config/loader.rb
|
|
656
678
|
- lib/labkit/rate_limit/config/parser.rb
|
|
657
679
|
- lib/labkit/rate_limit/config/proto/config_pb.rb
|
|
658
680
|
- lib/labkit/rate_limit/config/schema.rb
|