gitlab-labkit 5.12.0 → 5.14.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 5f286c7029701485334c9e4f78b26e799a0d74a016e64d7fd7bb881dfb82a853
4
- data.tar.gz: 8ee6361b4da8dda630297bed4526623f6b53ca813feb2ce2ebc2a5ea4522b1cd
3
+ metadata.gz: f9f95ff9d39b86e127ce7de851abe149fbe3659a31f4d82152c447884b59c89f
4
+ data.tar.gz: 70b231b5067e628000b51a031d83967d0ca1a9d9b43a2ab61ea72ad8d043fed7
5
5
  SHA512:
6
- metadata.gz: cd0a8fd90cee39271ac936e277ff72dde660db5b9d6615860142572a53fd2a7fbda90019b9fad4e0dae62c2c9d20656ae7ef905d588e9bea481b7f206bf890bd
7
- data.tar.gz: 79e35c32ea2e8e01e0b3df319fe563cbbe43986e2dc2446dd02220c60545d75b8fc2c4a393b1a9eea852b18af8bfb800b3aba3e641c43b2e401cf649cb1d7fea
6
+ metadata.gz: 2d8a4ebc1085ffe38412ee762b03703e730a2ebc16e7ccd7af4bd41a89e1720134c588f3d16a7035c9ce5b9723be39bd8dd014fdb0af90d58bd53b453dbeb6b2
7
+ data.tar.gz: 9d2b7897ae3d834ba335a4927b259e5ff4fd7d4bef23e610d4a90ab523629e50471844fe47214021c152e787b3997b0970283ce98055dca1e470ab9fcb9e9d1f
data/lib/labkit/fields.rb CHANGED
@@ -37,6 +37,11 @@ module Labkit
37
37
  # Detailed error message (e.g. "undefined method 'boom!' for nil").
38
38
  ERROR_MESSAGE = "error_message"
39
39
 
40
+ # Stack trace or backtrace captured when an error or panic is logged (e.g.
41
+ # by LabKit's panic-recovery middleware), as a single newline-separated
42
+ # string.
43
+ STACK_TRACE = "stack_trace"
44
+
40
45
  # HTTP response status code.
41
46
  HTTP_STATUS_CODE = "status"
42
47
 
@@ -172,6 +177,7 @@ module Labkit
172
177
  DUO_WORKFLOW_ID => 0,
173
178
  ERROR_TYPE => 0,
174
179
  ERROR_MESSAGE => 1,
180
+ STACK_TRACE => 0,
175
181
  HTTP_STATUS_CODE => 1,
176
182
  HTTP_METHOD => 0,
177
183
  HTTP_URL => 0,
@@ -229,6 +235,7 @@ module Labkit
229
235
  Fields::GL_USER_ID => %w[user_id userid extra.user_id extra.current_user_id meta.user_id gitlab_user_id],
230
236
  Fields::GL_USER_NAME => %w[username extra.user meta.user],
231
237
  Fields::ERROR_MESSAGE => %w[error err error.message exception.message graphql_errors],
238
+ Fields::STACK_TRACE => %w[backtrace],
232
239
  Fields::HTTP_STATUS_CODE => %w[status_code extra.status status_text http_status],
233
240
  Fields::HTTP_URL => %w[req_url],
234
241
  Fields::DURATION_S => %w[duration duration_ms elapsed_time actual_duration time_ms total_time gitaly.duration],
@@ -724,7 +724,7 @@ A gem schema that lacks or changes a vendored field raises `Labkit::RateLimit::C
724
724
  So upgrade the `gitlab-ratelimit-config-v1` gem in the application first, then the `gitlab-labkit` release that updates the vendored copy.
725
725
 
726
726
  Loading runs in four steps, `parse`, `schema`, `contract` and `rule`, and a failure raises `Labkit::RateLimit::Config::Error` whose `#step` names the step.
727
- All four steps exist; the registry that applies a document to a `Limiter` follows in [Part 3 of the plan](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/30263).
727
+ All four steps exist, and `Limiter` applies a document; see below.
728
728
 
729
729
  ```ruby
730
730
  doc = Labkit::RateLimit::Config::Parser.parse(
@@ -769,7 +769,7 @@ A rule built in Ruby keeps Ruby equality.
769
769
  A file rule checks its name in every environment and compiles `re` and `nre` with the `re2` gem, so a pattern that only Ruby accepts, such as a lookbehind, is rejected at step `rule`, and `^` anchors the whole text as it does in Go and Rust.
770
770
  `Document.empty` is a document with no limiters, for a host that has no contract yet.
771
771
 
772
- `scripts/vendor-ratelimit-proto.sh` also copies `config.proto` and `constraint.proto` next to the generated code, and `schema_spec.rb` parses them and compares every field rule (caps, bounds, presets, flags), every message rule id and message, and the name pattern with `Schema`, so a `PROTOS_REF` bump that changes any of those fails the suite until `Schema` follows.
772
+ `scripts/vendor-ratelimit-proto.sh` also copies `config.proto` and `constraint.proto` next to the generated code, and `schema_spec.rb` parses them and compares every field rule (caps, bounds, presets, flags), every message rule id and message, and the name pattern with `Schema`, and checks that message rules are reported in the order the proto declares them, so a `PROTOS_REF` bump that changes any of those fails the suite until `Schema` follows.
773
773
  The CEL expressions themselves are mirrored by hand and pinned by the examples.
774
774
 
775
775
  The parse step converts each YAML scalar by the type of the field it fills, so `eq: off`, `eq: True` and `eq: 9970` stay strings, `limit: "5"` becomes `5`, `action: "1"` becomes `ACTION_LIMIT`, and a YAML null leaves the field unset.
@@ -779,6 +779,35 @@ Each of these errors names the YAML line.
779
779
  An unknown field or an unknown enum name, such as `action: ACTION_LIMT`, fails in strict mode with its YAML line.
780
780
  Lenient mode ignores an unknown field and leaves an unknown enum name unset.
781
781
 
782
+ ### Applying a document to a limiter
783
+
784
+ A host loads one document at boot and makes it the default for every limiter:
785
+
786
+ ```ruby
787
+ Labkit::RateLimit.configure { |c| c.rule_document = doc }
788
+
789
+ limiter = Labkit::RateLimit::Limiter.new(
790
+ name: "rack_request",
791
+ guard_rules: [bypass_skip], # code rules that always run first; no override can replace one
792
+ rules: [protected_paths_claim, :contract, api_throttle] # the host's default list; :contract places the contract's default rules
793
+ )
794
+ limiter.rules # => the effective list: guards, then the document's override rules, then the defaults
795
+ ```
796
+
797
+ The effective order per limiter is: guard rules, then the document's override rules for that limiter, then the host's default list with the document's default rules at the `:contract` marker, or after the code defaults when there is none.
798
+ `rules:` holds `Rule` objects and at most one `:contract` marker; anything else raises `ArgumentError` at build time.
799
+ Until a host sets `rule_document`, it is `nil`: every limiter runs its code rules only, the marker places nothing, and `Labkit::RateLimit::Config` is never loaded.
800
+ A limiter reads the document once, when it is built, so set `rule_document` before any limiter is built, and rebuild the limiters to pick up a reloaded document.
801
+ A `:contract` marker that places nothing is logged once at build time: `rate_limit_contract_marker_without_document` when no document is set, `rate_limit_contract_marker_unknown_limiter` when the contract has no entry under the limiter's name.
802
+ An override with the same name as a default replaces it, in either list, and keeps its counter, because the Redis key is the limiter and the rule name.
803
+ An override that takes a guard rule's name is dropped and logged as `rate_limit_override_of_guard_rule`.
804
+ An override of a protected rule is dropped and logged as `rate_limit_override_of_protected_rule`; the loader already rejects it, and this keeps the answer the same for any other override source.
805
+ A contract default that shares a name with a code default raises in dev and test; in production the code default stays and the contract copy is logged as `rate_limit_contract_default_shadowed`, so a migration left half done is visible.
806
+ Names are compared as they will run, so an override replaces a code rule that `Limiter` renames at build time.
807
+ `limiter.rules` is the list the evaluator runs, after name sanitising and duplicate dropping.
808
+ `Labkit::RateLimit.check`, the throw-away helper, runs the rules it is given and reads no document.
809
+ A limiter built with no document, or with `Document.empty`, has today's rule list unchanged.
810
+
782
811
  ## Dev/test vs production guards
783
812
 
784
813
  `Limiter.new` and `Rule.new` validate names and configuration. In
@@ -15,9 +15,11 @@ module Labkit
15
15
  #
16
16
  # FIELD_RULES and MESSAGE_RULES are copied from the proto by hand, and
17
17
  # schema_spec.rb compares both with the vendored config.proto and
18
- # constraint.proto field by field, so a PROTOS_REF bump that changes a
19
- # cap, a flag, a rule id or a message fails the suite until this file
20
- # follows. The CEL expressions have no such check; the examples pin them.
18
+ # constraint.proto field by field, and checks that the checker reports
19
+ # message rules in declaration order, so a PROTOS_REF bump that changes a
20
+ # cap, a flag, a rule id, a message or the rule order fails the suite
21
+ # until this file follows. The CEL expressions have no such check; the
22
+ # examples pin them.
21
23
  #
22
24
  # The check is the strict reading of the schema. Lenient loading (Loader)
23
25
  # drops a rule whose matcher has no known operator before this step,
@@ -3,7 +3,11 @@
3
3
  module Labkit
4
4
  module RateLimit
5
5
  class Configuration
6
- attr_accessor :redis, :logger
6
+ # rule_document is the Document every Limiter reads its contract
7
+ # defaults and file overrides from, unless one is passed to Limiter.new.
8
+ # It stays nil until the host loads one, so a host without config files
9
+ # never loads Config and its vendored protobuf schema.
10
+ attr_accessor :redis, :logger, :rule_document
7
11
  end
8
12
  end
9
13
  end
@@ -18,14 +18,31 @@ module Labkit
18
18
  # render_429 if result.action == :block
19
19
  class Limiter
20
20
  NAME_PATTERN = /\A[a-z0-9_]+\z/
21
+ # Where in +rules:+ the contract's default rules go.
22
+ CONTRACT = :contract
21
23
 
22
- def initialize(name:, rules:, redis: nil, logger: nil)
24
+ # The rules the evaluator runs, in order: guard rules, the document's
25
+ # override rules for this limiter, then the default list, with names
26
+ # sanitised and duplicates dropped as prepare_rules does.
27
+ attr_reader :rules
28
+
29
+ # @param rules [Array<Rule, :contract>] the host's default rules, in
30
+ # order. The CONTRACT marker places the document's default rules for
31
+ # this limiter; without a marker they follow the code defaults.
32
+ # @param guard_rules [Array<Rule>] rules built in code that run before
33
+ # every other tier and that no override can replace.
34
+ # @param document [Config::Document, nil] the loaded contract and
35
+ # overrides; defaults to the configured one. With none, only the code
36
+ # rules run and the CONTRACT marker places nothing.
37
+ def initialize(name:, rules:, guard_rules: [], redis: nil, logger: nil, document: RateLimit.config.rule_document)
23
38
  @logger = logger || RateLimit.config.logger || Labkit::Logging::JsonLogger.new($stdout)
24
39
  @name = validate_name!(name)
40
+ effective = effective_rules(validate_entries!(guard_rules, "guard_rules"), validate_entries!(rules, "rules", marker: true), document)
41
+ @rules = prepare_rules(effective).freeze
25
42
 
26
43
  @evaluator = Evaluator.new(
27
44
  name: @name,
28
- rules: prepare_rules(rules),
45
+ rules: @rules,
29
46
  redis: redis || RateLimit.config.redis,
30
47
  logger: @logger
31
48
  )
@@ -98,6 +115,88 @@ module Labkit
98
115
  identifier.is_a?(Identifier) ? identifier : Identifier.new(identifier)
99
116
  end
100
117
 
118
+ # Both lists are typed by hand. A stray entry such as :contrat has a
119
+ # #name in Ruby 3+, so it would pass prepare_rules, reach the evaluator
120
+ # as a rule and fail the limiter open. Only rules: may carry the marker.
121
+ def validate_entries!(entries, label, marker: false)
122
+ markers = entries.count(CONTRACT)
123
+ raise ArgumentError, "#{label} may hold the #{CONTRACT.inspect} marker once, got #{markers}" if marker && markers > 1
124
+
125
+ entries.each do |entry|
126
+ next if entry.is_a?(Rule) || (marker && entry == CONTRACT)
127
+
128
+ allowed = marker ? "Rule objects or the #{CONTRACT.inspect} marker" : "Rule objects"
129
+ raise ArgumentError, "#{label} must hold #{allowed}, got #{entry.inspect}"
130
+ end
131
+ end
132
+
133
+ # Guard, override, default. An override with a default's name replaces
134
+ # that default and keeps its counter, because the Redis key is the
135
+ # limiter and rule name. A guard cannot be replaced: an override with a
136
+ # guard's name is dropped and logged, so the guard keeps running first.
137
+ def effective_rules(guard_rules, default_rules, document)
138
+ code_defaults = default_rules.reject { |entry| entry == CONTRACT }
139
+ warn_unplaced_marker(document) if default_rules.include?(CONTRACT)
140
+ return guard_rules + code_defaults if document.nil?
141
+
142
+ overrides = admitted_overrides(document, run_names(guard_rules))
143
+ override_names = run_names(overrides)
144
+ contract_defaults = contract_defaults_for(document, run_names(code_defaults))
145
+
146
+ defaults = default_rules.flat_map { |entry| entry == CONTRACT ? contract_defaults : [entry] }
147
+ defaults += contract_defaults unless default_rules.include?(CONTRACT)
148
+
149
+ guard_rules + overrides + defaults.reject { |rule| override_names.include?(sanitized_name(rule.name.to_s)) }
150
+ end
151
+
152
+ # The loader already refuses an override of a protected rule; this
153
+ # check stays so a later override source gets the same answer.
154
+ def admitted_overrides(document, guard_names)
155
+ protected_names = document.protected_rules(@name)
156
+ document.override_rules(@name).reject do |rule|
157
+ if guard_names.include?(rule.name)
158
+ @logger.warn(message: "rate_limit_override_of_guard_rule", limiter: @name, rule: rule.name)
159
+ elsif protected_names.include?(rule.name)
160
+ @logger.warn(message: "rate_limit_override_of_protected_rule", limiter: @name, rule: rule.name)
161
+ else
162
+ next false
163
+ end
164
+
165
+ true
166
+ end
167
+ end
168
+
169
+ # A rule defined both in code and in the contract is a migration left
170
+ # half done. Dev and test fail the build; production keeps the code
171
+ # rule, which is what ran before, and says so.
172
+ def contract_defaults_for(document, code_names)
173
+ document.default_rules(@name).reject do |rule|
174
+ next false unless code_names.include?(rule.name)
175
+
176
+ raise ArgumentError, "rule #{rule.name.inspect} of limiter #{@name} is defined in code and in the contract; remove one" if Labkit.dev_or_test?
177
+
178
+ @logger.warn(message: "rate_limit_contract_default_shadowed", limiter: @name, rule: rule.name)
179
+ true
180
+ end
181
+ end
182
+
183
+ # A marker that places nothing is a host mistake worth one line: the
184
+ # limiter was built before the document was set, or under a name the
185
+ # contract does not declare.
186
+ def warn_unplaced_marker(document)
187
+ if document.nil?
188
+ @logger.warn(message: "rate_limit_contract_marker_without_document", limiter: @name)
189
+ elsif !document.limiter?(@name)
190
+ @logger.warn(message: "rate_limit_contract_marker_unknown_limiter", limiter: @name)
191
+ end
192
+ end
193
+
194
+ # Names as they will run, so a code rule that prepare_rules renames
195
+ # still compares with the file rule meant to replace it.
196
+ def run_names(rules)
197
+ rules.map { |rule| sanitized_name(rule.name.to_s) }
198
+ end
199
+
101
200
  def validate_name!(name)
102
201
  raise ArgumentError, "name must be a non-empty String" unless name.is_a?(String) && !name.empty?
103
202
  return name if NAME_PATTERN.match?(name)
@@ -136,10 +235,9 @@ module Labkit
136
235
 
137
236
  def sanitize_rule_name(name)
138
237
  s = name.to_s
139
- return s if RULE_NAME_PATTERN.match?(s) && s.length <= RULE_NAME_MAX_LENGTH
238
+ sanitized = sanitized_name(s)
239
+ return s if sanitized == s
140
240
 
141
- sanitized = s.downcase.gsub(/[^a-z0-9_]/, "_")[0, RULE_NAME_MAX_LENGTH]
142
- sanitized = "unnamed_rule" if sanitized.empty?
143
241
  @logger.warn(
144
242
  message: "rate_limit_invalid_rule_name",
145
243
  original_name: s,
@@ -147,6 +245,15 @@ module Labkit
147
245
  )
148
246
  sanitized
149
247
  end
248
+
249
+ # The name a rule runs under, without the warning, for comparisons that
250
+ # happen before prepare_rules logs it once.
251
+ def sanitized_name(name)
252
+ return name if RULE_NAME_PATTERN.match?(name) && name.length <= RULE_NAME_MAX_LENGTH
253
+
254
+ sanitized = name.downcase.gsub(/[^a-z0-9_]/, "_")[0, RULE_NAME_MAX_LENGTH]
255
+ sanitized.empty? ? "unnamed_rule" : sanitized
256
+ end
150
257
  end
151
258
  end
152
259
  end
@@ -51,7 +51,9 @@ module Labkit
51
51
  # callables on rule limit/period; see Limiter#check
52
52
  # @return [Result]
53
53
  def check(name:, identifier:, rules:, redis: nil, logger: nil, cost: 1, rule_context: nil)
54
- Limiter.new(name: name, rules: rules, redis: redis, logger: logger)
54
+ # A throw-away limiter runs the rules it is given; the configured rule
55
+ # document applies to limiters a host builds and keeps.
56
+ Limiter.new(name: name, rules: rules, redis: redis, logger: logger, document: nil)
55
57
  .check(identifier, cost: cost, rule_context: rule_context)
56
58
  end
57
59
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: gitlab-labkit
3
3
  version: !ruby/object:Gem::Version
4
- version: 5.12.0
4
+ version: 5.14.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Andrew Newdigate