gitlab-labkit 5.12.0 → 5.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/lib/labkit/rate_limit/README.md +30 -1
- data/lib/labkit/rate_limit/configuration.rb +5 -1
- data/lib/labkit/rate_limit/limiter.rb +112 -5
- data/lib/labkit/rate_limit.rb +3 -1
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: d23f73a2fd1a445640d1eb5b139b37c7663b4071080fb81599e581e8e694d0b4
|
|
4
|
+
data.tar.gz: 1ae6ecb899e6eccba3c0852b5a580c7a90dab5c987798322131e1ccca28b34d7
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: f6f1c85a2fb3fdc4d3e5884aa342722057597a844b9a4641c802f244abbf8ea928b308c554d49141165e7723854dfdcfbb829dc6f70ef2dbfd5b38c773081304
|
|
7
|
+
data.tar.gz: f1d8f12623bbfb1226cec7691cb883ea184b0e04ba66243347f687fa2366aadfb750f751ebdda2c6bde65c8731aa47727fbeb079f614c955a039801acab3bcad
|
|
@@ -724,7 +724,7 @@ A gem schema that lacks or changes a vendored field raises `Labkit::RateLimit::C
|
|
|
724
724
|
So upgrade the `gitlab-ratelimit-config-v1` gem in the application first, then the `gitlab-labkit` release that updates the vendored copy.
|
|
725
725
|
|
|
726
726
|
Loading runs in four steps, `parse`, `schema`, `contract` and `rule`, and a failure raises `Labkit::RateLimit::Config::Error` whose `#step` names the step.
|
|
727
|
-
All four steps exist
|
|
727
|
+
All four steps exist, and `Limiter` applies a document; see below.
|
|
728
728
|
|
|
729
729
|
```ruby
|
|
730
730
|
doc = Labkit::RateLimit::Config::Parser.parse(
|
|
@@ -779,6 +779,35 @@ Each of these errors names the YAML line.
|
|
|
779
779
|
An unknown field or an unknown enum name, such as `action: ACTION_LIMT`, fails in strict mode with its YAML line.
|
|
780
780
|
Lenient mode ignores an unknown field and leaves an unknown enum name unset.
|
|
781
781
|
|
|
782
|
+
### Applying a document to a limiter
|
|
783
|
+
|
|
784
|
+
A host loads one document at boot and makes it the default for every limiter:
|
|
785
|
+
|
|
786
|
+
```ruby
|
|
787
|
+
Labkit::RateLimit.configure { |c| c.rule_document = doc }
|
|
788
|
+
|
|
789
|
+
limiter = Labkit::RateLimit::Limiter.new(
|
|
790
|
+
name: "rack_request",
|
|
791
|
+
guard_rules: [bypass_skip], # code rules that always run first; no override can replace one
|
|
792
|
+
rules: [protected_paths_claim, :contract, api_throttle] # the host's default list; :contract places the contract's default rules
|
|
793
|
+
)
|
|
794
|
+
limiter.rules # => the effective list: guards, then the document's override rules, then the defaults
|
|
795
|
+
```
|
|
796
|
+
|
|
797
|
+
The effective order per limiter is: guard rules, then the document's override rules for that limiter, then the host's default list with the document's default rules at the `:contract` marker, or after the code defaults when there is none.
|
|
798
|
+
`rules:` holds `Rule` objects and at most one `:contract` marker; anything else raises `ArgumentError` at build time.
|
|
799
|
+
Until a host sets `rule_document`, it is `nil`: every limiter runs its code rules only, the marker places nothing, and `Labkit::RateLimit::Config` is never loaded.
|
|
800
|
+
A limiter reads the document once, when it is built, so set `rule_document` before any limiter is built, and rebuild the limiters to pick up a reloaded document.
|
|
801
|
+
A `:contract` marker that places nothing is logged once at build time: `rate_limit_contract_marker_without_document` when no document is set, `rate_limit_contract_marker_unknown_limiter` when the contract has no entry under the limiter's name.
|
|
802
|
+
An override with the same name as a default replaces it, in either list, and keeps its counter, because the Redis key is the limiter and the rule name.
|
|
803
|
+
An override that takes a guard rule's name is dropped and logged as `rate_limit_override_of_guard_rule`.
|
|
804
|
+
An override of a protected rule is dropped and logged as `rate_limit_override_of_protected_rule`; the loader already rejects it, and this keeps the answer the same for any other override source.
|
|
805
|
+
A contract default that shares a name with a code default raises in dev and test; in production the code default stays and the contract copy is logged as `rate_limit_contract_default_shadowed`, so a migration left half done is visible.
|
|
806
|
+
Names are compared as they will run, so an override replaces a code rule that `Limiter` renames at build time.
|
|
807
|
+
`limiter.rules` is the list the evaluator runs, after name sanitising and duplicate dropping.
|
|
808
|
+
`Labkit::RateLimit.check`, the throw-away helper, runs the rules it is given and reads no document.
|
|
809
|
+
A limiter built with no document, or with `Document.empty`, has today's rule list unchanged.
|
|
810
|
+
|
|
782
811
|
## Dev/test vs production guards
|
|
783
812
|
|
|
784
813
|
`Limiter.new` and `Rule.new` validate names and configuration. In
|
|
@@ -3,7 +3,11 @@
|
|
|
3
3
|
module Labkit
|
|
4
4
|
module RateLimit
|
|
5
5
|
class Configuration
|
|
6
|
-
|
|
6
|
+
# rule_document is the Document every Limiter reads its contract
|
|
7
|
+
# defaults and file overrides from, unless one is passed to Limiter.new.
|
|
8
|
+
# It stays nil until the host loads one, so a host without config files
|
|
9
|
+
# never loads Config and its vendored protobuf schema.
|
|
10
|
+
attr_accessor :redis, :logger, :rule_document
|
|
7
11
|
end
|
|
8
12
|
end
|
|
9
13
|
end
|
|
@@ -18,14 +18,31 @@ module Labkit
|
|
|
18
18
|
# render_429 if result.action == :block
|
|
19
19
|
class Limiter
|
|
20
20
|
NAME_PATTERN = /\A[a-z0-9_]+\z/
|
|
21
|
+
# Where in +rules:+ the contract's default rules go.
|
|
22
|
+
CONTRACT = :contract
|
|
21
23
|
|
|
22
|
-
|
|
24
|
+
# The rules the evaluator runs, in order: guard rules, the document's
|
|
25
|
+
# override rules for this limiter, then the default list, with names
|
|
26
|
+
# sanitised and duplicates dropped as prepare_rules does.
|
|
27
|
+
attr_reader :rules
|
|
28
|
+
|
|
29
|
+
# @param rules [Array<Rule, :contract>] the host's default rules, in
|
|
30
|
+
# order. The CONTRACT marker places the document's default rules for
|
|
31
|
+
# this limiter; without a marker they follow the code defaults.
|
|
32
|
+
# @param guard_rules [Array<Rule>] rules built in code that run before
|
|
33
|
+
# every other tier and that no override can replace.
|
|
34
|
+
# @param document [Config::Document, nil] the loaded contract and
|
|
35
|
+
# overrides; defaults to the configured one. With none, only the code
|
|
36
|
+
# rules run and the CONTRACT marker places nothing.
|
|
37
|
+
def initialize(name:, rules:, guard_rules: [], redis: nil, logger: nil, document: RateLimit.config.rule_document)
|
|
23
38
|
@logger = logger || RateLimit.config.logger || Labkit::Logging::JsonLogger.new($stdout)
|
|
24
39
|
@name = validate_name!(name)
|
|
40
|
+
effective = effective_rules(validate_entries!(guard_rules, "guard_rules"), validate_entries!(rules, "rules", marker: true), document)
|
|
41
|
+
@rules = prepare_rules(effective).freeze
|
|
25
42
|
|
|
26
43
|
@evaluator = Evaluator.new(
|
|
27
44
|
name: @name,
|
|
28
|
-
rules:
|
|
45
|
+
rules: @rules,
|
|
29
46
|
redis: redis || RateLimit.config.redis,
|
|
30
47
|
logger: @logger
|
|
31
48
|
)
|
|
@@ -98,6 +115,88 @@ module Labkit
|
|
|
98
115
|
identifier.is_a?(Identifier) ? identifier : Identifier.new(identifier)
|
|
99
116
|
end
|
|
100
117
|
|
|
118
|
+
# Both lists are typed by hand. A stray entry such as :contrat has a
|
|
119
|
+
# #name in Ruby 3+, so it would pass prepare_rules, reach the evaluator
|
|
120
|
+
# as a rule and fail the limiter open. Only rules: may carry the marker.
|
|
121
|
+
def validate_entries!(entries, label, marker: false)
|
|
122
|
+
markers = entries.count(CONTRACT)
|
|
123
|
+
raise ArgumentError, "#{label} may hold the #{CONTRACT.inspect} marker once, got #{markers}" if marker && markers > 1
|
|
124
|
+
|
|
125
|
+
entries.each do |entry|
|
|
126
|
+
next if entry.is_a?(Rule) || (marker && entry == CONTRACT)
|
|
127
|
+
|
|
128
|
+
allowed = marker ? "Rule objects or the #{CONTRACT.inspect} marker" : "Rule objects"
|
|
129
|
+
raise ArgumentError, "#{label} must hold #{allowed}, got #{entry.inspect}"
|
|
130
|
+
end
|
|
131
|
+
end
|
|
132
|
+
|
|
133
|
+
# Guard, override, default. An override with a default's name replaces
|
|
134
|
+
# that default and keeps its counter, because the Redis key is the
|
|
135
|
+
# limiter and rule name. A guard cannot be replaced: an override with a
|
|
136
|
+
# guard's name is dropped and logged, so the guard keeps running first.
|
|
137
|
+
def effective_rules(guard_rules, default_rules, document)
|
|
138
|
+
code_defaults = default_rules.reject { |entry| entry == CONTRACT }
|
|
139
|
+
warn_unplaced_marker(document) if default_rules.include?(CONTRACT)
|
|
140
|
+
return guard_rules + code_defaults if document.nil?
|
|
141
|
+
|
|
142
|
+
overrides = admitted_overrides(document, run_names(guard_rules))
|
|
143
|
+
override_names = run_names(overrides)
|
|
144
|
+
contract_defaults = contract_defaults_for(document, run_names(code_defaults))
|
|
145
|
+
|
|
146
|
+
defaults = default_rules.flat_map { |entry| entry == CONTRACT ? contract_defaults : [entry] }
|
|
147
|
+
defaults += contract_defaults unless default_rules.include?(CONTRACT)
|
|
148
|
+
|
|
149
|
+
guard_rules + overrides + defaults.reject { |rule| override_names.include?(sanitized_name(rule.name.to_s)) }
|
|
150
|
+
end
|
|
151
|
+
|
|
152
|
+
# The loader already refuses an override of a protected rule; this
|
|
153
|
+
# check stays so a later override source gets the same answer.
|
|
154
|
+
def admitted_overrides(document, guard_names)
|
|
155
|
+
protected_names = document.protected_rules(@name)
|
|
156
|
+
document.override_rules(@name).reject do |rule|
|
|
157
|
+
if guard_names.include?(rule.name)
|
|
158
|
+
@logger.warn(message: "rate_limit_override_of_guard_rule", limiter: @name, rule: rule.name)
|
|
159
|
+
elsif protected_names.include?(rule.name)
|
|
160
|
+
@logger.warn(message: "rate_limit_override_of_protected_rule", limiter: @name, rule: rule.name)
|
|
161
|
+
else
|
|
162
|
+
next false
|
|
163
|
+
end
|
|
164
|
+
|
|
165
|
+
true
|
|
166
|
+
end
|
|
167
|
+
end
|
|
168
|
+
|
|
169
|
+
# A rule defined both in code and in the contract is a migration left
|
|
170
|
+
# half done. Dev and test fail the build; production keeps the code
|
|
171
|
+
# rule, which is what ran before, and says so.
|
|
172
|
+
def contract_defaults_for(document, code_names)
|
|
173
|
+
document.default_rules(@name).reject do |rule|
|
|
174
|
+
next false unless code_names.include?(rule.name)
|
|
175
|
+
|
|
176
|
+
raise ArgumentError, "rule #{rule.name.inspect} of limiter #{@name} is defined in code and in the contract; remove one" if Labkit.dev_or_test?
|
|
177
|
+
|
|
178
|
+
@logger.warn(message: "rate_limit_contract_default_shadowed", limiter: @name, rule: rule.name)
|
|
179
|
+
true
|
|
180
|
+
end
|
|
181
|
+
end
|
|
182
|
+
|
|
183
|
+
# A marker that places nothing is a host mistake worth one line: the
|
|
184
|
+
# limiter was built before the document was set, or under a name the
|
|
185
|
+
# contract does not declare.
|
|
186
|
+
def warn_unplaced_marker(document)
|
|
187
|
+
if document.nil?
|
|
188
|
+
@logger.warn(message: "rate_limit_contract_marker_without_document", limiter: @name)
|
|
189
|
+
elsif !document.limiter?(@name)
|
|
190
|
+
@logger.warn(message: "rate_limit_contract_marker_unknown_limiter", limiter: @name)
|
|
191
|
+
end
|
|
192
|
+
end
|
|
193
|
+
|
|
194
|
+
# Names as they will run, so a code rule that prepare_rules renames
|
|
195
|
+
# still compares with the file rule meant to replace it.
|
|
196
|
+
def run_names(rules)
|
|
197
|
+
rules.map { |rule| sanitized_name(rule.name.to_s) }
|
|
198
|
+
end
|
|
199
|
+
|
|
101
200
|
def validate_name!(name)
|
|
102
201
|
raise ArgumentError, "name must be a non-empty String" unless name.is_a?(String) && !name.empty?
|
|
103
202
|
return name if NAME_PATTERN.match?(name)
|
|
@@ -136,10 +235,9 @@ module Labkit
|
|
|
136
235
|
|
|
137
236
|
def sanitize_rule_name(name)
|
|
138
237
|
s = name.to_s
|
|
139
|
-
|
|
238
|
+
sanitized = sanitized_name(s)
|
|
239
|
+
return s if sanitized == s
|
|
140
240
|
|
|
141
|
-
sanitized = s.downcase.gsub(/[^a-z0-9_]/, "_")[0, RULE_NAME_MAX_LENGTH]
|
|
142
|
-
sanitized = "unnamed_rule" if sanitized.empty?
|
|
143
241
|
@logger.warn(
|
|
144
242
|
message: "rate_limit_invalid_rule_name",
|
|
145
243
|
original_name: s,
|
|
@@ -147,6 +245,15 @@ module Labkit
|
|
|
147
245
|
)
|
|
148
246
|
sanitized
|
|
149
247
|
end
|
|
248
|
+
|
|
249
|
+
# The name a rule runs under, without the warning, for comparisons that
|
|
250
|
+
# happen before prepare_rules logs it once.
|
|
251
|
+
def sanitized_name(name)
|
|
252
|
+
return name if RULE_NAME_PATTERN.match?(name) && name.length <= RULE_NAME_MAX_LENGTH
|
|
253
|
+
|
|
254
|
+
sanitized = name.downcase.gsub(/[^a-z0-9_]/, "_")[0, RULE_NAME_MAX_LENGTH]
|
|
255
|
+
sanitized.empty? ? "unnamed_rule" : sanitized
|
|
256
|
+
end
|
|
150
257
|
end
|
|
151
258
|
end
|
|
152
259
|
end
|
data/lib/labkit/rate_limit.rb
CHANGED
|
@@ -51,7 +51,9 @@ module Labkit
|
|
|
51
51
|
# callables on rule limit/period; see Limiter#check
|
|
52
52
|
# @return [Result]
|
|
53
53
|
def check(name:, identifier:, rules:, redis: nil, logger: nil, cost: 1, rule_context: nil)
|
|
54
|
-
|
|
54
|
+
# A throw-away limiter runs the rules it is given; the configured rule
|
|
55
|
+
# document applies to limiters a host builds and keeps.
|
|
56
|
+
Limiter.new(name: name, rules: rules, redis: redis, logger: logger, document: nil)
|
|
55
57
|
.check(identifier, cost: cost, rule_context: rule_context)
|
|
56
58
|
end
|
|
57
59
|
end
|