gitlab-labkit 5.12.0 → 5.13.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 5f286c7029701485334c9e4f78b26e799a0d74a016e64d7fd7bb881dfb82a853
4
- data.tar.gz: 8ee6361b4da8dda630297bed4526623f6b53ca813feb2ce2ebc2a5ea4522b1cd
3
+ metadata.gz: d23f73a2fd1a445640d1eb5b139b37c7663b4071080fb81599e581e8e694d0b4
4
+ data.tar.gz: 1ae6ecb899e6eccba3c0852b5a580c7a90dab5c987798322131e1ccca28b34d7
5
5
  SHA512:
6
- metadata.gz: cd0a8fd90cee39271ac936e277ff72dde660db5b9d6615860142572a53fd2a7fbda90019b9fad4e0dae62c2c9d20656ae7ef905d588e9bea481b7f206bf890bd
7
- data.tar.gz: 79e35c32ea2e8e01e0b3df319fe563cbbe43986e2dc2446dd02220c60545d75b8fc2c4a393b1a9eea852b18af8bfb800b3aba3e641c43b2e401cf649cb1d7fea
6
+ metadata.gz: f6f1c85a2fb3fdc4d3e5884aa342722057597a844b9a4641c802f244abbf8ea928b308c554d49141165e7723854dfdcfbb829dc6f70ef2dbfd5b38c773081304
7
+ data.tar.gz: f1d8f12623bbfb1226cec7691cb883ea184b0e04ba66243347f687fa2366aadfb750f751ebdda2c6bde65c8731aa47727fbeb079f614c955a039801acab3bcad
@@ -724,7 +724,7 @@ A gem schema that lacks or changes a vendored field raises `Labkit::RateLimit::C
724
724
  So upgrade the `gitlab-ratelimit-config-v1` gem in the application first, then the `gitlab-labkit` release that updates the vendored copy.
725
725
 
726
726
  Loading runs in four steps, `parse`, `schema`, `contract` and `rule`, and a failure raises `Labkit::RateLimit::Config::Error` whose `#step` names the step.
727
- All four steps exist; the registry that applies a document to a `Limiter` follows in [Part 3 of the plan](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/30263).
727
+ All four steps exist, and `Limiter` applies a document; see below.
728
728
 
729
729
  ```ruby
730
730
  doc = Labkit::RateLimit::Config::Parser.parse(
@@ -779,6 +779,35 @@ Each of these errors names the YAML line.
779
779
  An unknown field or an unknown enum name, such as `action: ACTION_LIMT`, fails in strict mode with its YAML line.
780
780
  Lenient mode ignores an unknown field and leaves an unknown enum name unset.
781
781
 
782
+ ### Applying a document to a limiter
783
+
784
+ A host loads one document at boot and makes it the default for every limiter:
785
+
786
+ ```ruby
787
+ Labkit::RateLimit.configure { |c| c.rule_document = doc }
788
+
789
+ limiter = Labkit::RateLimit::Limiter.new(
790
+ name: "rack_request",
791
+ guard_rules: [bypass_skip], # code rules that always run first; no override can replace one
792
+ rules: [protected_paths_claim, :contract, api_throttle] # the host's default list; :contract places the contract's default rules
793
+ )
794
+ limiter.rules # => the effective list: guards, then the document's override rules, then the defaults
795
+ ```
796
+
797
+ The effective order per limiter is: guard rules, then the document's override rules for that limiter, then the host's default list with the document's default rules at the `:contract` marker, or after the code defaults when there is none.
798
+ `rules:` holds `Rule` objects and at most one `:contract` marker; anything else raises `ArgumentError` at build time.
799
+ Until a host sets `rule_document`, it is `nil`: every limiter runs its code rules only, the marker places nothing, and `Labkit::RateLimit::Config` is never loaded.
800
+ A limiter reads the document once, when it is built, so set `rule_document` before any limiter is built, and rebuild the limiters to pick up a reloaded document.
801
+ A `:contract` marker that places nothing is logged once at build time: `rate_limit_contract_marker_without_document` when no document is set, `rate_limit_contract_marker_unknown_limiter` when the contract has no entry under the limiter's name.
802
+ An override with the same name as a default replaces it, in either list, and keeps its counter, because the Redis key is the limiter and the rule name.
803
+ An override that takes a guard rule's name is dropped and logged as `rate_limit_override_of_guard_rule`.
804
+ An override of a protected rule is dropped and logged as `rate_limit_override_of_protected_rule`; the loader already rejects it, and this keeps the answer the same for any other override source.
805
+ A contract default that shares a name with a code default raises in dev and test; in production the code default stays and the contract copy is logged as `rate_limit_contract_default_shadowed`, so a migration left half done is visible.
806
+ Names are compared as they will run, so an override replaces a code rule that `Limiter` renames at build time.
807
+ `limiter.rules` is the list the evaluator runs, after name sanitising and duplicate dropping.
808
+ `Labkit::RateLimit.check`, the throw-away helper, runs the rules it is given and reads no document.
809
+ A limiter built with no document, or with `Document.empty`, has today's rule list unchanged.
810
+
782
811
  ## Dev/test vs production guards
783
812
 
784
813
  `Limiter.new` and `Rule.new` validate names and configuration. In
@@ -3,7 +3,11 @@
3
3
  module Labkit
4
4
  module RateLimit
5
5
  class Configuration
6
- attr_accessor :redis, :logger
6
+ # rule_document is the Document every Limiter reads its contract
7
+ # defaults and file overrides from, unless one is passed to Limiter.new.
8
+ # It stays nil until the host loads one, so a host without config files
9
+ # never loads Config and its vendored protobuf schema.
10
+ attr_accessor :redis, :logger, :rule_document
7
11
  end
8
12
  end
9
13
  end
@@ -18,14 +18,31 @@ module Labkit
18
18
  # render_429 if result.action == :block
19
19
  class Limiter
20
20
  NAME_PATTERN = /\A[a-z0-9_]+\z/
21
+ # Where in +rules:+ the contract's default rules go.
22
+ CONTRACT = :contract
21
23
 
22
- def initialize(name:, rules:, redis: nil, logger: nil)
24
+ # The rules the evaluator runs, in order: guard rules, the document's
25
+ # override rules for this limiter, then the default list, with names
26
+ # sanitised and duplicates dropped as prepare_rules does.
27
+ attr_reader :rules
28
+
29
+ # @param rules [Array<Rule, :contract>] the host's default rules, in
30
+ # order. The CONTRACT marker places the document's default rules for
31
+ # this limiter; without a marker they follow the code defaults.
32
+ # @param guard_rules [Array<Rule>] rules built in code that run before
33
+ # every other tier and that no override can replace.
34
+ # @param document [Config::Document, nil] the loaded contract and
35
+ # overrides; defaults to the configured one. With none, only the code
36
+ # rules run and the CONTRACT marker places nothing.
37
+ def initialize(name:, rules:, guard_rules: [], redis: nil, logger: nil, document: RateLimit.config.rule_document)
23
38
  @logger = logger || RateLimit.config.logger || Labkit::Logging::JsonLogger.new($stdout)
24
39
  @name = validate_name!(name)
40
+ effective = effective_rules(validate_entries!(guard_rules, "guard_rules"), validate_entries!(rules, "rules", marker: true), document)
41
+ @rules = prepare_rules(effective).freeze
25
42
 
26
43
  @evaluator = Evaluator.new(
27
44
  name: @name,
28
- rules: prepare_rules(rules),
45
+ rules: @rules,
29
46
  redis: redis || RateLimit.config.redis,
30
47
  logger: @logger
31
48
  )
@@ -98,6 +115,88 @@ module Labkit
98
115
  identifier.is_a?(Identifier) ? identifier : Identifier.new(identifier)
99
116
  end
100
117
 
118
+ # Both lists are typed by hand. A stray entry such as :contrat has a
119
+ # #name in Ruby 3+, so it would pass prepare_rules, reach the evaluator
120
+ # as a rule and fail the limiter open. Only rules: may carry the marker.
121
+ def validate_entries!(entries, label, marker: false)
122
+ markers = entries.count(CONTRACT)
123
+ raise ArgumentError, "#{label} may hold the #{CONTRACT.inspect} marker once, got #{markers}" if marker && markers > 1
124
+
125
+ entries.each do |entry|
126
+ next if entry.is_a?(Rule) || (marker && entry == CONTRACT)
127
+
128
+ allowed = marker ? "Rule objects or the #{CONTRACT.inspect} marker" : "Rule objects"
129
+ raise ArgumentError, "#{label} must hold #{allowed}, got #{entry.inspect}"
130
+ end
131
+ end
132
+
133
+ # Guard, override, default. An override with a default's name replaces
134
+ # that default and keeps its counter, because the Redis key is the
135
+ # limiter and rule name. A guard cannot be replaced: an override with a
136
+ # guard's name is dropped and logged, so the guard keeps running first.
137
+ def effective_rules(guard_rules, default_rules, document)
138
+ code_defaults = default_rules.reject { |entry| entry == CONTRACT }
139
+ warn_unplaced_marker(document) if default_rules.include?(CONTRACT)
140
+ return guard_rules + code_defaults if document.nil?
141
+
142
+ overrides = admitted_overrides(document, run_names(guard_rules))
143
+ override_names = run_names(overrides)
144
+ contract_defaults = contract_defaults_for(document, run_names(code_defaults))
145
+
146
+ defaults = default_rules.flat_map { |entry| entry == CONTRACT ? contract_defaults : [entry] }
147
+ defaults += contract_defaults unless default_rules.include?(CONTRACT)
148
+
149
+ guard_rules + overrides + defaults.reject { |rule| override_names.include?(sanitized_name(rule.name.to_s)) }
150
+ end
151
+
152
+ # The loader already refuses an override of a protected rule; this
153
+ # check stays so a later override source gets the same answer.
154
+ def admitted_overrides(document, guard_names)
155
+ protected_names = document.protected_rules(@name)
156
+ document.override_rules(@name).reject do |rule|
157
+ if guard_names.include?(rule.name)
158
+ @logger.warn(message: "rate_limit_override_of_guard_rule", limiter: @name, rule: rule.name)
159
+ elsif protected_names.include?(rule.name)
160
+ @logger.warn(message: "rate_limit_override_of_protected_rule", limiter: @name, rule: rule.name)
161
+ else
162
+ next false
163
+ end
164
+
165
+ true
166
+ end
167
+ end
168
+
169
+ # A rule defined both in code and in the contract is a migration left
170
+ # half done. Dev and test fail the build; production keeps the code
171
+ # rule, which is what ran before, and says so.
172
+ def contract_defaults_for(document, code_names)
173
+ document.default_rules(@name).reject do |rule|
174
+ next false unless code_names.include?(rule.name)
175
+
176
+ raise ArgumentError, "rule #{rule.name.inspect} of limiter #{@name} is defined in code and in the contract; remove one" if Labkit.dev_or_test?
177
+
178
+ @logger.warn(message: "rate_limit_contract_default_shadowed", limiter: @name, rule: rule.name)
179
+ true
180
+ end
181
+ end
182
+
183
+ # A marker that places nothing is a host mistake worth one line: the
184
+ # limiter was built before the document was set, or under a name the
185
+ # contract does not declare.
186
+ def warn_unplaced_marker(document)
187
+ if document.nil?
188
+ @logger.warn(message: "rate_limit_contract_marker_without_document", limiter: @name)
189
+ elsif !document.limiter?(@name)
190
+ @logger.warn(message: "rate_limit_contract_marker_unknown_limiter", limiter: @name)
191
+ end
192
+ end
193
+
194
+ # Names as they will run, so a code rule that prepare_rules renames
195
+ # still compares with the file rule meant to replace it.
196
+ def run_names(rules)
197
+ rules.map { |rule| sanitized_name(rule.name.to_s) }
198
+ end
199
+
101
200
  def validate_name!(name)
102
201
  raise ArgumentError, "name must be a non-empty String" unless name.is_a?(String) && !name.empty?
103
202
  return name if NAME_PATTERN.match?(name)
@@ -136,10 +235,9 @@ module Labkit
136
235
 
137
236
  def sanitize_rule_name(name)
138
237
  s = name.to_s
139
- return s if RULE_NAME_PATTERN.match?(s) && s.length <= RULE_NAME_MAX_LENGTH
238
+ sanitized = sanitized_name(s)
239
+ return s if sanitized == s
140
240
 
141
- sanitized = s.downcase.gsub(/[^a-z0-9_]/, "_")[0, RULE_NAME_MAX_LENGTH]
142
- sanitized = "unnamed_rule" if sanitized.empty?
143
241
  @logger.warn(
144
242
  message: "rate_limit_invalid_rule_name",
145
243
  original_name: s,
@@ -147,6 +245,15 @@ module Labkit
147
245
  )
148
246
  sanitized
149
247
  end
248
+
249
+ # The name a rule runs under, without the warning, for comparisons that
250
+ # happen before prepare_rules logs it once.
251
+ def sanitized_name(name)
252
+ return name if RULE_NAME_PATTERN.match?(name) && name.length <= RULE_NAME_MAX_LENGTH
253
+
254
+ sanitized = name.downcase.gsub(/[^a-z0-9_]/, "_")[0, RULE_NAME_MAX_LENGTH]
255
+ sanitized.empty? ? "unnamed_rule" : sanitized
256
+ end
150
257
  end
151
258
  end
152
259
  end
@@ -51,7 +51,9 @@ module Labkit
51
51
  # callables on rule limit/period; see Limiter#check
52
52
  # @return [Result]
53
53
  def check(name:, identifier:, rules:, redis: nil, logger: nil, cost: 1, rule_context: nil)
54
- Limiter.new(name: name, rules: rules, redis: redis, logger: logger)
54
+ # A throw-away limiter runs the rules it is given; the configured rule
55
+ # document applies to limiters a host builds and keeps.
56
+ Limiter.new(name: name, rules: rules, redis: redis, logger: logger, document: nil)
55
57
  .check(identifier, cost: cost, rule_context: rule_context)
56
58
  end
57
59
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: gitlab-labkit
3
3
  version: !ruby/object:Gem::Version
4
- version: 5.12.0
4
+ version: 5.13.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Andrew Newdigate