gitlab-labkit 5.11.0 → 5.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/gitlab-labkit.gemspec +1 -0
- data/lib/labkit/rate_limit/README.md +55 -2
- data/lib/labkit/rate_limit/config/document.rb +64 -0
- data/lib/labkit/rate_limit/config/loader.rb +246 -0
- data/lib/labkit/rate_limit/config/schema.rb +3 -3
- data/lib/labkit/rate_limit/config.rb +2 -0
- data/lib/labkit/rate_limit/configuration.rb +5 -1
- data/lib/labkit/rate_limit/evaluator.rb +21 -1
- data/lib/labkit/rate_limit/limiter.rb +112 -5
- data/lib/labkit/rate_limit/matcher.rb +44 -10
- data/lib/labkit/rate_limit/rule.rb +28 -12
- data/lib/labkit/rate_limit.rb +3 -1
- metadata +23 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: d23f73a2fd1a445640d1eb5b139b37c7663b4071080fb81599e581e8e694d0b4
|
|
4
|
+
data.tar.gz: 1ae6ecb899e6eccba3c0852b5a580c7a90dab5c987798322131e1ccca28b34d7
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: f6f1c85a2fb3fdc4d3e5884aa342722057597a844b9a4641c802f244abbf8ea928b308c554d49141165e7723854dfdcfbb829dc6f70ef2dbfd5b38c773081304
|
|
7
|
+
data.tar.gz: f1d8f12623bbfb1226cec7691cb883ea184b0e04ba66243347f687fa2366aadfb750f751ebdda2c6bde65c8731aa47727fbeb079f614c955a039801acab3bcad
|
data/gitlab-labkit.gemspec
CHANGED
|
@@ -36,6 +36,7 @@ Gem::Specification.new do |spec|
|
|
|
36
36
|
spec.add_runtime_dependency "opentracing", ">= 0.4", "< 1"
|
|
37
37
|
spec.add_runtime_dependency "pg_query", ">= 6.1.0", "< 7.0"
|
|
38
38
|
spec.add_runtime_dependency "prometheus-client-mmap", ">= 1.2", "< 2.0"
|
|
39
|
+
spec.add_runtime_dependency "re2", ">= 2.15", "< 3"
|
|
39
40
|
spec.add_runtime_dependency "redis", "> 3.0.0", "< 6.0.0"
|
|
40
41
|
spec.add_runtime_dependency "thrift", "< 0.25" # Pinned for jaeger-client 1.1.0 compatibility; thrift 0.25 requires transports to implement message_boundaries?
|
|
41
42
|
|
|
@@ -724,7 +724,7 @@ A gem schema that lacks or changes a vendored field raises `Labkit::RateLimit::C
|
|
|
724
724
|
So upgrade the `gitlab-ratelimit-config-v1` gem in the application first, then the `gitlab-labkit` release that updates the vendored copy.
|
|
725
725
|
|
|
726
726
|
Loading runs in four steps, `parse`, `schema`, `contract` and `rule`, and a failure raises `Labkit::RateLimit::Config::Error` whose `#step` names the step.
|
|
727
|
-
|
|
727
|
+
All four steps exist, and `Limiter` applies a document; see below.
|
|
728
728
|
|
|
729
729
|
```ruby
|
|
730
730
|
doc = Labkit::RateLimit::Config::Parser.parse(
|
|
@@ -744,7 +744,31 @@ Inside one message the order is protovalidate's: message rules, then the oneof,
|
|
|
744
744
|
A protobuf map keeps no document order and Go iterates it in an unspecified order, so Ruby reports map entries by sorted key and a cross-SDK comparison matches violations per entry, not by position.
|
|
745
745
|
Map keys in a path are quoted as Go's `strconv.Quote` prints them.
|
|
746
746
|
The check is the strict reading of the schema: a matcher whose operator this binary does not know fails `required`.
|
|
747
|
-
Lenient loading, through the
|
|
747
|
+
Lenient loading, through the `Loader` below, drops such a rule with a warning before the schema step, so a newer operator does not reject the file.
|
|
748
|
+
`Labkit::RateLimit::Config::Loader` runs all four steps and returns a `Document`:
|
|
749
|
+
|
|
750
|
+
```ruby
|
|
751
|
+
loader = Labkit::RateLimit::Config::Loader.new(strict: Rails.env.test?, logger: Gitlab::AppLogger)
|
|
752
|
+
doc = loader.load(contracts: Dir["config/rate_limits/*.yaml"], rate_limits: "config/rate_limits.yaml")
|
|
753
|
+
doc.limiter?("rack_request") # => true
|
|
754
|
+
doc.available_properties("rack_request") # => [:plan, :path, :requester_id]
|
|
755
|
+
doc.default_rules("rack_request") # => [Rule, ...] from the contract, document order
|
|
756
|
+
doc.override_rules("rack_request") # => [Rule, ...] from the rate limits file
|
|
757
|
+
doc.warnings # => [{ step:, limiter:, rule:, message: }, ...]
|
|
758
|
+
```
|
|
759
|
+
|
|
760
|
+
The contract step merges the contract files: a limiter is declared, with its description, in exactly one file, a later file may add properties and append default rules, and every rule may use only the properties its limiter lists.
|
|
761
|
+
It rejects an override that takes a protected name or carries a `position`, and warns when an override drops a match key of the default it replaces.
|
|
762
|
+
A contract mistake rejects in both modes, because the contract is the application's own.
|
|
763
|
+
In lenient mode a rule for an unknown limiter, with an unknown property, with a protected name, or with a matcher whose only operator this version does not know is dropped with a warning, and a position is ignored; any other failure rejects the whole rate limits file.
|
|
764
|
+
Every warning is also logged as `rate_limit_config_warning` with `step`, `limiter`, `rule` and `detail`.
|
|
765
|
+
|
|
766
|
+
The rule step builds each proto rule as a `Rule` with `source: :file`.
|
|
767
|
+
A file rule compares identifier values as strings, as every SDK does for a rule from a document: `{ eq: "false" }` matches a `false` fact, a missing key reads as `""`, an integer reads as its decimal digits and a symbol as its name; other value types fall back to `to_s` and have no fixed cross-SDK form, so a host keeps them out of identifiers that file rules match on.
|
|
768
|
+
A rule built in Ruby keeps Ruby equality.
|
|
769
|
+
A file rule checks its name in every environment and compiles `re` and `nre` with the `re2` gem, so a pattern that only Ruby accepts, such as a lookbehind, is rejected at step `rule`, and `^` anchors the whole text as it does in Go and Rust.
|
|
770
|
+
`Document.empty` is a document with no limiters, for a host that has no contract yet.
|
|
771
|
+
|
|
748
772
|
`scripts/vendor-ratelimit-proto.sh` also copies `config.proto` and `constraint.proto` next to the generated code, and `schema_spec.rb` parses them and compares every field rule (caps, bounds, presets, flags), every message rule id and message, and the name pattern with `Schema`, so a `PROTOS_REF` bump that changes any of those fails the suite until `Schema` follows.
|
|
749
773
|
The CEL expressions themselves are mirrored by hand and pinned by the examples.
|
|
750
774
|
|
|
@@ -755,6 +779,35 @@ Each of these errors names the YAML line.
|
|
|
755
779
|
An unknown field or an unknown enum name, such as `action: ACTION_LIMT`, fails in strict mode with its YAML line.
|
|
756
780
|
Lenient mode ignores an unknown field and leaves an unknown enum name unset.
|
|
757
781
|
|
|
782
|
+
### Applying a document to a limiter
|
|
783
|
+
|
|
784
|
+
A host loads one document at boot and makes it the default for every limiter:
|
|
785
|
+
|
|
786
|
+
```ruby
|
|
787
|
+
Labkit::RateLimit.configure { |c| c.rule_document = doc }
|
|
788
|
+
|
|
789
|
+
limiter = Labkit::RateLimit::Limiter.new(
|
|
790
|
+
name: "rack_request",
|
|
791
|
+
guard_rules: [bypass_skip], # code rules that always run first; no override can replace one
|
|
792
|
+
rules: [protected_paths_claim, :contract, api_throttle] # the host's default list; :contract places the contract's default rules
|
|
793
|
+
)
|
|
794
|
+
limiter.rules # => the effective list: guards, then the document's override rules, then the defaults
|
|
795
|
+
```
|
|
796
|
+
|
|
797
|
+
The effective order per limiter is: guard rules, then the document's override rules for that limiter, then the host's default list with the document's default rules at the `:contract` marker, or after the code defaults when there is none.
|
|
798
|
+
`rules:` holds `Rule` objects and at most one `:contract` marker; anything else raises `ArgumentError` at build time.
|
|
799
|
+
Until a host sets `rule_document`, it is `nil`: every limiter runs its code rules only, the marker places nothing, and `Labkit::RateLimit::Config` is never loaded.
|
|
800
|
+
A limiter reads the document once, when it is built, so set `rule_document` before any limiter is built, and rebuild the limiters to pick up a reloaded document.
|
|
801
|
+
A `:contract` marker that places nothing is logged once at build time: `rate_limit_contract_marker_without_document` when no document is set, `rate_limit_contract_marker_unknown_limiter` when the contract has no entry under the limiter's name.
|
|
802
|
+
An override with the same name as a default replaces it, in either list, and keeps its counter, because the Redis key is the limiter and the rule name.
|
|
803
|
+
An override that takes a guard rule's name is dropped and logged as `rate_limit_override_of_guard_rule`.
|
|
804
|
+
An override of a protected rule is dropped and logged as `rate_limit_override_of_protected_rule`; the loader already rejects it, and this keeps the answer the same for any other override source.
|
|
805
|
+
A contract default that shares a name with a code default raises in dev and test; in production the code default stays and the contract copy is logged as `rate_limit_contract_default_shadowed`, so a migration left half done is visible.
|
|
806
|
+
Names are compared as they will run, so an override replaces a code rule that `Limiter` renames at build time.
|
|
807
|
+
`limiter.rules` is the list the evaluator runs, after name sanitising and duplicate dropping.
|
|
808
|
+
`Labkit::RateLimit.check`, the throw-away helper, runs the rules it is given and reads no document.
|
|
809
|
+
A limiter built with no document, or with `Document.empty`, has today's rule list unchanged.
|
|
810
|
+
|
|
758
811
|
## Dev/test vs production guards
|
|
759
812
|
|
|
760
813
|
`Limiter.new` and `Rule.new` validate names and configuration. In
|
|
@@ -0,0 +1,64 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Labkit
|
|
4
|
+
module RateLimit
|
|
5
|
+
module Config
|
|
6
|
+
# Document is what Loader#load returns: the merged contract and the rules
|
|
7
|
+
# of the rate limits file, per limiter, as Rule objects a Limiter can use.
|
|
8
|
+
# It is immutable; a host builds a new one to reload.
|
|
9
|
+
class Document
|
|
10
|
+
# One limiter's slice of the document.
|
|
11
|
+
Limiter = Data.define(:properties, :default_rules, :override_rules, :protected_rules)
|
|
12
|
+
|
|
13
|
+
attr_reader :warnings
|
|
14
|
+
|
|
15
|
+
# A document with no limiters: every host default applies unchanged.
|
|
16
|
+
def self.empty
|
|
17
|
+
new({}, [])
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
# @param limiters [Hash{String => Limiter}]
|
|
21
|
+
# @param warnings [Array<Hash>] with :step, :limiter, :rule, :message
|
|
22
|
+
def initialize(limiters, warnings)
|
|
23
|
+
@limiters = limiters.freeze
|
|
24
|
+
@warnings = warnings.freeze
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
def limiter?(name)
|
|
28
|
+
@limiters.key?(name.to_s)
|
|
29
|
+
end
|
|
30
|
+
|
|
31
|
+
# @return [Array<String>] every limiter the contract declares
|
|
32
|
+
def limiters
|
|
33
|
+
@limiters.keys
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
# @return [Array<Symbol>] the properties a rule of this limiter may use
|
|
37
|
+
def available_properties(name)
|
|
38
|
+
slice(name)&.properties || []
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
# @return [Array<Rule>] the file rules for this limiter, document order
|
|
42
|
+
def override_rules(name)
|
|
43
|
+
slice(name)&.override_rules || []
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
# @return [Array<Rule>] the contract's default rules, document order
|
|
47
|
+
def default_rules(name)
|
|
48
|
+
slice(name)&.default_rules || []
|
|
49
|
+
end
|
|
50
|
+
|
|
51
|
+
# @return [Array<String>] default rule names no override may replace
|
|
52
|
+
def protected_rules(name)
|
|
53
|
+
slice(name)&.protected_rules || []
|
|
54
|
+
end
|
|
55
|
+
|
|
56
|
+
private
|
|
57
|
+
|
|
58
|
+
def slice(name)
|
|
59
|
+
@limiters[name.to_s]
|
|
60
|
+
end
|
|
61
|
+
end
|
|
62
|
+
end
|
|
63
|
+
end
|
|
64
|
+
end
|
|
@@ -0,0 +1,246 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Labkit
|
|
4
|
+
module RateLimit
|
|
5
|
+
module Config
|
|
6
|
+
# Loader runs the four steps of the config loader over a set of contract
|
|
7
|
+
# files and an optional rate limits file, and returns a Document.
|
|
8
|
+
#
|
|
9
|
+
# parse and schema are Parser and Schema. The contract step lives here: it
|
|
10
|
+
# merges the contract files, checks that every rule uses only properties
|
|
11
|
+
# its limiter lists, that no override takes a protected name, and that a
|
|
12
|
+
# file rule carries no position. The rule step converts each proto Rule
|
|
13
|
+
# into a Rule with source: :file, which compiles patterns with RE2.
|
|
14
|
+
#
|
|
15
|
+
# Strict mode, for CI, rejects on any failure. Lenient mode, for a running
|
|
16
|
+
# process, drops with a warning a rule for an unknown limiter, with an
|
|
17
|
+
# unknown property, with a protected name, or with a matcher whose only
|
|
18
|
+
# operator this binary does not know, and ignores a position. Any other
|
|
19
|
+
# failure rejects the whole file: the host then logs and runs on its
|
|
20
|
+
# defaults. Contract files are the application's own, so a contract
|
|
21
|
+
# mistake rejects in both modes.
|
|
22
|
+
class Loader
|
|
23
|
+
ACTIONS = { ACTION_LIMIT: :limit, ACTION_LOG: :log, ACTION_SKIP: :skip }.freeze
|
|
24
|
+
# The operators this version converts. A newer protos gem can decode
|
|
25
|
+
# an operator or an action outside these sets.
|
|
26
|
+
OPERATORS = %i[eq ne re nre one_of none_of all].freeze
|
|
27
|
+
|
|
28
|
+
# Merged view of one limiter across the contract files.
|
|
29
|
+
Spec = Struct.new(:properties, :default_rules, :protected_rules, :described)
|
|
30
|
+
|
|
31
|
+
# Default rules and properties per limiter, across every contract
|
|
32
|
+
# file. Schema caps one file's list; the merge must hold the same line.
|
|
33
|
+
MAX_DEFAULT_RULES = Schema::FIELD_RULES.fetch("LimiterSpec.default_rules").fetch(:max_items)
|
|
34
|
+
MAX_PROPERTIES = Schema::FIELD_RULES.fetch("LimiterSpec.available_properties").fetch(:max_items)
|
|
35
|
+
|
|
36
|
+
# @param strict [Boolean] reject on any failure instead of dropping rules
|
|
37
|
+
# @param logger [#warn, nil] receives each warning as it is recorded
|
|
38
|
+
def initialize(strict: false, logger: nil)
|
|
39
|
+
@strict = strict
|
|
40
|
+
@logger = logger || RateLimit.config.logger || Labkit::Logging::JsonLogger.new($stdout)
|
|
41
|
+
# load keeps the run's warnings on the instance, so two threads
|
|
42
|
+
# sharing one Loader would mix them.
|
|
43
|
+
@mutex = Mutex.new
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
# @param contracts [Array<String>] paths of available_limiters files, in order
|
|
47
|
+
# @param rate_limits [String, nil] path of the rate_limits file
|
|
48
|
+
# @return [Document]
|
|
49
|
+
# @raise [Error] with the step that failed
|
|
50
|
+
def load(contracts:, rate_limits: nil)
|
|
51
|
+
@mutex.synchronize { load_locked(contracts, rate_limits) }
|
|
52
|
+
end
|
|
53
|
+
|
|
54
|
+
private
|
|
55
|
+
|
|
56
|
+
attr_reader :strict
|
|
57
|
+
|
|
58
|
+
def load_locked(contracts, rate_limits)
|
|
59
|
+
@warnings = []
|
|
60
|
+
specs = merge_contracts(contracts)
|
|
61
|
+
overrides = rate_limits ? load_rate_limits(rate_limits, specs) : {}
|
|
62
|
+
|
|
63
|
+
limiters = specs.to_h do |name, spec|
|
|
64
|
+
defaults = spec.default_rules.map { |rule| convert(name, rule) }
|
|
65
|
+
[name, Document::Limiter.new(spec.properties.freeze, defaults.freeze, overrides.fetch(name, []).freeze, spec.protected_rules.freeze)]
|
|
66
|
+
end
|
|
67
|
+
Document.new(limiters, @warnings)
|
|
68
|
+
end
|
|
69
|
+
|
|
70
|
+
def merge_contracts(paths)
|
|
71
|
+
specs = {}
|
|
72
|
+
paths.each do |path|
|
|
73
|
+
doc = Schema.check!(Parser.parse(path, Proto::AvailableLimiters, strict: true))
|
|
74
|
+
doc.available_limiters.each { |name, proto_spec| merge_limiter(specs, name, proto_spec, path) }
|
|
75
|
+
end
|
|
76
|
+
specs.each do |name, spec|
|
|
77
|
+
contract_error("limiter #{name} has no description in any contract file") unless spec.described
|
|
78
|
+
check_caps!(name, spec)
|
|
79
|
+
spec.default_rules.each { |rule| check_properties!(name, rule, spec) }
|
|
80
|
+
end
|
|
81
|
+
specs
|
|
82
|
+
end
|
|
83
|
+
|
|
84
|
+
# A limiter is declared, with its description, in exactly one file. A
|
|
85
|
+
# later file may add properties and append default rules.
|
|
86
|
+
def merge_limiter(specs, name, proto_spec, path)
|
|
87
|
+
spec = specs[name] ||= Spec.new([], [], [], false)
|
|
88
|
+
unless proto_spec.description.empty?
|
|
89
|
+
contract_error("limiter #{name} is declared twice, again in #{path}") if spec.described
|
|
90
|
+
spec.described = true
|
|
91
|
+
end
|
|
92
|
+
|
|
93
|
+
proto_spec.available_properties.each { |p| spec.properties << p.to_sym unless spec.properties.include?(p.to_sym) }
|
|
94
|
+
proto_spec.default_rules.each do |rule|
|
|
95
|
+
contract_error("limiter #{name}: default rule #{rule.name} is defined twice, again in #{path}") if spec.default_rules.any? { |r| r.name == rule.name }
|
|
96
|
+
spec.default_rules << rule
|
|
97
|
+
end
|
|
98
|
+
spec.protected_rules.concat(proto_spec.protected_rules.to_a - spec.protected_rules)
|
|
99
|
+
end
|
|
100
|
+
|
|
101
|
+
def load_rate_limits(path, specs)
|
|
102
|
+
doc = Parser.parse(path, Proto::RateLimits, strict: strict)
|
|
103
|
+
drop_unknown_operators(doc) unless strict
|
|
104
|
+
Schema.check!(doc)
|
|
105
|
+
|
|
106
|
+
# Map#to_h ignores a block, so the overrides are collected by hand.
|
|
107
|
+
overrides = {}
|
|
108
|
+
doc.rate_limits.each do |name, list|
|
|
109
|
+
spec = specs[name]
|
|
110
|
+
if spec
|
|
111
|
+
overrides[name] = list.rules.filter_map { |rule| override_for(name, rule, spec) }
|
|
112
|
+
else
|
|
113
|
+
contract_error("rate_limits names limiter #{name}, which no contract declares") if strict
|
|
114
|
+
warn(:contract, name, "", "limiter #{name} is not in the contract; its rules are dropped")
|
|
115
|
+
end
|
|
116
|
+
end
|
|
117
|
+
overrides
|
|
118
|
+
end
|
|
119
|
+
|
|
120
|
+
# An unknown operator is an empty oneof after the parse step, which the
|
|
121
|
+
# schema step would reject as a missing operator for the whole file.
|
|
122
|
+
def drop_unknown_operators(doc)
|
|
123
|
+
doc.rate_limits.each do |name, list|
|
|
124
|
+
kept = list.rules.reject do |rule|
|
|
125
|
+
dropped = rule.match.any? { |_, matcher| unknown_operator?(matcher) }
|
|
126
|
+
warn(:parse, name, rule.name, "a matcher uses an operator this version does not know; rule dropped") if dropped
|
|
127
|
+
dropped
|
|
128
|
+
end
|
|
129
|
+
list.rules.replace(kept)
|
|
130
|
+
end
|
|
131
|
+
end
|
|
132
|
+
|
|
133
|
+
def unknown_operator?(matcher)
|
|
134
|
+
return true unless OPERATORS.include?(matcher.op)
|
|
135
|
+
|
|
136
|
+
matcher.op == :all && matcher.all.matchers.any? { |m| unknown_operator?(m) }
|
|
137
|
+
end
|
|
138
|
+
|
|
139
|
+
# The contract checks on one file rule. Returns the Rule, or nil when
|
|
140
|
+
# lenient mode drops it.
|
|
141
|
+
def override_for(name, rule, spec)
|
|
142
|
+
return reject(:contract, name, rule.name, "#{rule.name} is a protected rule of #{name}; the contract alone sets its limit") if spec.protected_rules.include?(rule.name)
|
|
143
|
+
|
|
144
|
+
unknown = unknown_property(rule, spec)
|
|
145
|
+
return reject(:contract, name, rule.name, "property #{unknown} is not listed for limiter #{name}") if unknown
|
|
146
|
+
|
|
147
|
+
if rule.has_position?
|
|
148
|
+
contract_error("limiter #{name}: rule #{rule.name} has a position; only a published rule can") if strict
|
|
149
|
+
warn(:contract, name, rule.name, "position is ignored on a file rule")
|
|
150
|
+
end
|
|
151
|
+
|
|
152
|
+
warn_dropped_match_keys(name, rule, spec)
|
|
153
|
+
convert(name, rule)
|
|
154
|
+
end
|
|
155
|
+
|
|
156
|
+
# An override that matches less than the default it replaces widens the
|
|
157
|
+
# rule. Allowed, since every matching rule counts, but worth a line in
|
|
158
|
+
# the log.
|
|
159
|
+
def warn_dropped_match_keys(name, rule, spec)
|
|
160
|
+
default = spec.default_rules.find { |d| d.name == rule.name }
|
|
161
|
+
return unless default
|
|
162
|
+
|
|
163
|
+
dropped = default.match.keys - rule.match.keys
|
|
164
|
+
warn(:contract, name, rule.name, "dropped match keys: #{dropped.sort.join(', ')}") unless dropped.empty?
|
|
165
|
+
end
|
|
166
|
+
|
|
167
|
+
def check_caps!(name, spec)
|
|
168
|
+
rules = spec.default_rules.size
|
|
169
|
+
contract_error("limiter #{name} has #{rules} default rules across the contract files; at most #{MAX_DEFAULT_RULES} are allowed") if rules > MAX_DEFAULT_RULES
|
|
170
|
+
|
|
171
|
+
properties = spec.properties.size
|
|
172
|
+
contract_error("limiter #{name} lists #{properties} properties across the contract files; at most #{MAX_PROPERTIES} are allowed") if properties > MAX_PROPERTIES
|
|
173
|
+
end
|
|
174
|
+
|
|
175
|
+
def check_properties!(name, rule, spec)
|
|
176
|
+
unknown = unknown_property(rule, spec)
|
|
177
|
+
contract_error("limiter #{name}: default rule #{rule.name} uses property #{unknown}, which the limiter does not list") if unknown
|
|
178
|
+
end
|
|
179
|
+
|
|
180
|
+
def unknown_property(rule, spec)
|
|
181
|
+
(used_properties(rule) - spec.properties).first
|
|
182
|
+
end
|
|
183
|
+
|
|
184
|
+
def used_properties(rule)
|
|
185
|
+
used = rule.characteristics.map(&:to_sym) + rule.match.keys.map(&:to_sym)
|
|
186
|
+
used << rule.count_distinct.to_sym unless rule.count_distinct.empty?
|
|
187
|
+
used
|
|
188
|
+
end
|
|
189
|
+
|
|
190
|
+
def convert(limiter, rule)
|
|
191
|
+
Rule.new(
|
|
192
|
+
name: rule.name,
|
|
193
|
+
action: ACTIONS.fetch(rule.action) { raise ArgumentError, "unknown action #{rule.action.inspect}" },
|
|
194
|
+
limit: rule.has_limit? ? rule.limit : nil,
|
|
195
|
+
period: rule.has_period_s? ? rule.period_s : nil,
|
|
196
|
+
ban_for: rule.has_ban_for_s? ? rule.ban_for_s : nil,
|
|
197
|
+
characteristics: rule.characteristics.to_a,
|
|
198
|
+
count_distinct: rule.count_distinct.empty? ? nil : rule.count_distinct,
|
|
199
|
+
match: match_input(rule.match),
|
|
200
|
+
source: :file
|
|
201
|
+
)
|
|
202
|
+
rescue ArgumentError => e
|
|
203
|
+
raise Error.new(:rule, "#{limiter}/#{rule.name}: #{e.message}")
|
|
204
|
+
end
|
|
205
|
+
|
|
206
|
+
# Map#to_h would turn the nested messages into plain Hashes, so the
|
|
207
|
+
# entries are walked as messages.
|
|
208
|
+
def match_input(map)
|
|
209
|
+
input = {}
|
|
210
|
+
map.each { |key, matcher| input[key] = matcher_input(matcher) }
|
|
211
|
+
input
|
|
212
|
+
end
|
|
213
|
+
|
|
214
|
+
# The proto Matcher in the shape Rule.new accepts.
|
|
215
|
+
def matcher_input(matcher)
|
|
216
|
+
case matcher.op
|
|
217
|
+
when :eq, :ne, :re, :nre then { matcher.op => matcher[matcher.op.to_s] }
|
|
218
|
+
when :one_of then { oneOf: matcher.one_of.values.to_a }
|
|
219
|
+
when :none_of then { noneOf: matcher.none_of.values.to_a }
|
|
220
|
+
when :all then matcher.all.matchers.map { |m| matcher_input(m) }
|
|
221
|
+
else raise ArgumentError, "unknown matcher operator #{matcher.op.inspect}"
|
|
222
|
+
end
|
|
223
|
+
end
|
|
224
|
+
|
|
225
|
+
def reject(step, limiter, rule, message)
|
|
226
|
+
contract_error("limiter #{limiter}: rule #{rule}: #{message}") if strict
|
|
227
|
+
|
|
228
|
+
warn(step, limiter, rule, message)
|
|
229
|
+
nil
|
|
230
|
+
end
|
|
231
|
+
|
|
232
|
+
def warn(step, limiter, rule, message)
|
|
233
|
+
warning = { step: step, limiter: limiter, rule: rule, message: message }
|
|
234
|
+
@warnings << warning
|
|
235
|
+
# The warning's own :message is the detail; the log line keeps the
|
|
236
|
+
# fixed message every rate_limit log line has.
|
|
237
|
+
@logger.warn(message: "rate_limit_config_warning", step: step, limiter: limiter, rule: rule, detail: message)
|
|
238
|
+
end
|
|
239
|
+
|
|
240
|
+
def contract_error(message)
|
|
241
|
+
raise Error.new(:contract, message)
|
|
242
|
+
end
|
|
243
|
+
end
|
|
244
|
+
end
|
|
245
|
+
end
|
|
246
|
+
end
|
|
@@ -19,9 +19,9 @@ module Labkit
|
|
|
19
19
|
# cap, a flag, a rule id or a message fails the suite until this file
|
|
20
20
|
# follows. The CEL expressions have no such check; the examples pin them.
|
|
21
21
|
#
|
|
22
|
-
# The check is the strict reading of the schema. Lenient loading
|
|
23
|
-
# rule whose matcher has no known operator before this step
|
|
24
|
-
#
|
|
22
|
+
# The check is the strict reading of the schema. Lenient loading (Loader)
|
|
23
|
+
# drops a rule whose matcher has no known operator before this step,
|
|
24
|
+
# because such a matcher fails `required` here.
|
|
25
25
|
module Schema
|
|
26
26
|
# for_key marks a violation on a map key; its path is the entry's path.
|
|
27
27
|
Violation = Data.define(:path, :rule_id, :message, :for_key) do
|
|
@@ -16,6 +16,8 @@ module Labkit
|
|
|
16
16
|
Proto = Gitlab::Ratelimit::Config::V1
|
|
17
17
|
|
|
18
18
|
autoload :Error, "labkit/rate_limit/config/error"
|
|
19
|
+
autoload :Document, "labkit/rate_limit/config/document"
|
|
20
|
+
autoload :Loader, "labkit/rate_limit/config/loader"
|
|
19
21
|
autoload :Parser, "labkit/rate_limit/config/parser"
|
|
20
22
|
autoload :Schema, "labkit/rate_limit/config/schema"
|
|
21
23
|
end
|
|
@@ -3,7 +3,11 @@
|
|
|
3
3
|
module Labkit
|
|
4
4
|
module RateLimit
|
|
5
5
|
class Configuration
|
|
6
|
-
|
|
6
|
+
# rule_document is the Document every Limiter reads its contract
|
|
7
|
+
# defaults and file overrides from, unless one is passed to Limiter.new.
|
|
8
|
+
# It stays nil until the host loads one, so a host without config files
|
|
9
|
+
# never loads Config and its vendored protobuf schema.
|
|
10
|
+
attr_accessor :redis, :logger, :rule_document
|
|
7
11
|
end
|
|
8
12
|
end
|
|
9
13
|
end
|
|
@@ -321,8 +321,28 @@ module Labkit
|
|
|
321
321
|
cursor.rule = nil unless e
|
|
322
322
|
end
|
|
323
323
|
|
|
324
|
+
# A file rule compares as strings, the way every SDK does for a rule
|
|
325
|
+
# from a document. A code rule keeps Ruby equality, so {eq: true}
|
|
326
|
+
# matches true only.
|
|
324
327
|
def rule_matches?(rule, identifier)
|
|
325
|
-
rule.
|
|
328
|
+
if rule.source == :file
|
|
329
|
+
rule.match.all? { |key, matcher| matcher.match?(file_value(identifier[key])) }
|
|
330
|
+
else
|
|
331
|
+
rule.match.all? { |key, matcher| matcher.match?(identifier[key]) }
|
|
332
|
+
end
|
|
333
|
+
end
|
|
334
|
+
|
|
335
|
+
# The string form of an identifier value for a file rule. A missing key
|
|
336
|
+
# reads as "". Strings, symbols, booleans and integers have one form in
|
|
337
|
+
# every SDK; anything else falls back to to_s and is the host's to keep
|
|
338
|
+
# out of identifiers.
|
|
339
|
+
def file_value(value)
|
|
340
|
+
case value
|
|
341
|
+
when nil then ""
|
|
342
|
+
when String then value
|
|
343
|
+
when Symbol then value.name
|
|
344
|
+
else value.to_s
|
|
345
|
+
end
|
|
326
346
|
end
|
|
327
347
|
|
|
328
348
|
def missing_count_distinct_value?(rule, identifier)
|
|
@@ -18,14 +18,31 @@ module Labkit
|
|
|
18
18
|
# render_429 if result.action == :block
|
|
19
19
|
class Limiter
|
|
20
20
|
NAME_PATTERN = /\A[a-z0-9_]+\z/
|
|
21
|
+
# Where in +rules:+ the contract's default rules go.
|
|
22
|
+
CONTRACT = :contract
|
|
21
23
|
|
|
22
|
-
|
|
24
|
+
# The rules the evaluator runs, in order: guard rules, the document's
|
|
25
|
+
# override rules for this limiter, then the default list, with names
|
|
26
|
+
# sanitised and duplicates dropped as prepare_rules does.
|
|
27
|
+
attr_reader :rules
|
|
28
|
+
|
|
29
|
+
# @param rules [Array<Rule, :contract>] the host's default rules, in
|
|
30
|
+
# order. The CONTRACT marker places the document's default rules for
|
|
31
|
+
# this limiter; without a marker they follow the code defaults.
|
|
32
|
+
# @param guard_rules [Array<Rule>] rules built in code that run before
|
|
33
|
+
# every other tier and that no override can replace.
|
|
34
|
+
# @param document [Config::Document, nil] the loaded contract and
|
|
35
|
+
# overrides; defaults to the configured one. With none, only the code
|
|
36
|
+
# rules run and the CONTRACT marker places nothing.
|
|
37
|
+
def initialize(name:, rules:, guard_rules: [], redis: nil, logger: nil, document: RateLimit.config.rule_document)
|
|
23
38
|
@logger = logger || RateLimit.config.logger || Labkit::Logging::JsonLogger.new($stdout)
|
|
24
39
|
@name = validate_name!(name)
|
|
40
|
+
effective = effective_rules(validate_entries!(guard_rules, "guard_rules"), validate_entries!(rules, "rules", marker: true), document)
|
|
41
|
+
@rules = prepare_rules(effective).freeze
|
|
25
42
|
|
|
26
43
|
@evaluator = Evaluator.new(
|
|
27
44
|
name: @name,
|
|
28
|
-
rules:
|
|
45
|
+
rules: @rules,
|
|
29
46
|
redis: redis || RateLimit.config.redis,
|
|
30
47
|
logger: @logger
|
|
31
48
|
)
|
|
@@ -98,6 +115,88 @@ module Labkit
|
|
|
98
115
|
identifier.is_a?(Identifier) ? identifier : Identifier.new(identifier)
|
|
99
116
|
end
|
|
100
117
|
|
|
118
|
+
# Both lists are typed by hand. A stray entry such as :contrat has a
|
|
119
|
+
# #name in Ruby 3+, so it would pass prepare_rules, reach the evaluator
|
|
120
|
+
# as a rule and fail the limiter open. Only rules: may carry the marker.
|
|
121
|
+
def validate_entries!(entries, label, marker: false)
|
|
122
|
+
markers = entries.count(CONTRACT)
|
|
123
|
+
raise ArgumentError, "#{label} may hold the #{CONTRACT.inspect} marker once, got #{markers}" if marker && markers > 1
|
|
124
|
+
|
|
125
|
+
entries.each do |entry|
|
|
126
|
+
next if entry.is_a?(Rule) || (marker && entry == CONTRACT)
|
|
127
|
+
|
|
128
|
+
allowed = marker ? "Rule objects or the #{CONTRACT.inspect} marker" : "Rule objects"
|
|
129
|
+
raise ArgumentError, "#{label} must hold #{allowed}, got #{entry.inspect}"
|
|
130
|
+
end
|
|
131
|
+
end
|
|
132
|
+
|
|
133
|
+
# Guard, override, default. An override with a default's name replaces
|
|
134
|
+
# that default and keeps its counter, because the Redis key is the
|
|
135
|
+
# limiter and rule name. A guard cannot be replaced: an override with a
|
|
136
|
+
# guard's name is dropped and logged, so the guard keeps running first.
|
|
137
|
+
def effective_rules(guard_rules, default_rules, document)
|
|
138
|
+
code_defaults = default_rules.reject { |entry| entry == CONTRACT }
|
|
139
|
+
warn_unplaced_marker(document) if default_rules.include?(CONTRACT)
|
|
140
|
+
return guard_rules + code_defaults if document.nil?
|
|
141
|
+
|
|
142
|
+
overrides = admitted_overrides(document, run_names(guard_rules))
|
|
143
|
+
override_names = run_names(overrides)
|
|
144
|
+
contract_defaults = contract_defaults_for(document, run_names(code_defaults))
|
|
145
|
+
|
|
146
|
+
defaults = default_rules.flat_map { |entry| entry == CONTRACT ? contract_defaults : [entry] }
|
|
147
|
+
defaults += contract_defaults unless default_rules.include?(CONTRACT)
|
|
148
|
+
|
|
149
|
+
guard_rules + overrides + defaults.reject { |rule| override_names.include?(sanitized_name(rule.name.to_s)) }
|
|
150
|
+
end
|
|
151
|
+
|
|
152
|
+
# The loader already refuses an override of a protected rule; this
|
|
153
|
+
# check stays so a later override source gets the same answer.
|
|
154
|
+
def admitted_overrides(document, guard_names)
|
|
155
|
+
protected_names = document.protected_rules(@name)
|
|
156
|
+
document.override_rules(@name).reject do |rule|
|
|
157
|
+
if guard_names.include?(rule.name)
|
|
158
|
+
@logger.warn(message: "rate_limit_override_of_guard_rule", limiter: @name, rule: rule.name)
|
|
159
|
+
elsif protected_names.include?(rule.name)
|
|
160
|
+
@logger.warn(message: "rate_limit_override_of_protected_rule", limiter: @name, rule: rule.name)
|
|
161
|
+
else
|
|
162
|
+
next false
|
|
163
|
+
end
|
|
164
|
+
|
|
165
|
+
true
|
|
166
|
+
end
|
|
167
|
+
end
|
|
168
|
+
|
|
169
|
+
# A rule defined both in code and in the contract is a migration left
|
|
170
|
+
# half done. Dev and test fail the build; production keeps the code
|
|
171
|
+
# rule, which is what ran before, and says so.
|
|
172
|
+
def contract_defaults_for(document, code_names)
|
|
173
|
+
document.default_rules(@name).reject do |rule|
|
|
174
|
+
next false unless code_names.include?(rule.name)
|
|
175
|
+
|
|
176
|
+
raise ArgumentError, "rule #{rule.name.inspect} of limiter #{@name} is defined in code and in the contract; remove one" if Labkit.dev_or_test?
|
|
177
|
+
|
|
178
|
+
@logger.warn(message: "rate_limit_contract_default_shadowed", limiter: @name, rule: rule.name)
|
|
179
|
+
true
|
|
180
|
+
end
|
|
181
|
+
end
|
|
182
|
+
|
|
183
|
+
# A marker that places nothing is a host mistake worth one line: the
|
|
184
|
+
# limiter was built before the document was set, or under a name the
|
|
185
|
+
# contract does not declare.
|
|
186
|
+
def warn_unplaced_marker(document)
|
|
187
|
+
if document.nil?
|
|
188
|
+
@logger.warn(message: "rate_limit_contract_marker_without_document", limiter: @name)
|
|
189
|
+
elsif !document.limiter?(@name)
|
|
190
|
+
@logger.warn(message: "rate_limit_contract_marker_unknown_limiter", limiter: @name)
|
|
191
|
+
end
|
|
192
|
+
end
|
|
193
|
+
|
|
194
|
+
# Names as they will run, so a code rule that prepare_rules renames
|
|
195
|
+
# still compares with the file rule meant to replace it.
|
|
196
|
+
def run_names(rules)
|
|
197
|
+
rules.map { |rule| sanitized_name(rule.name.to_s) }
|
|
198
|
+
end
|
|
199
|
+
|
|
101
200
|
def validate_name!(name)
|
|
102
201
|
raise ArgumentError, "name must be a non-empty String" unless name.is_a?(String) && !name.empty?
|
|
103
202
|
return name if NAME_PATTERN.match?(name)
|
|
@@ -136,10 +235,9 @@ module Labkit
|
|
|
136
235
|
|
|
137
236
|
def sanitize_rule_name(name)
|
|
138
237
|
s = name.to_s
|
|
139
|
-
|
|
238
|
+
sanitized = sanitized_name(s)
|
|
239
|
+
return s if sanitized == s
|
|
140
240
|
|
|
141
|
-
sanitized = s.downcase.gsub(/[^a-z0-9_]/, "_")[0, RULE_NAME_MAX_LENGTH]
|
|
142
|
-
sanitized = "unnamed_rule" if sanitized.empty?
|
|
143
241
|
@logger.warn(
|
|
144
242
|
message: "rate_limit_invalid_rule_name",
|
|
145
243
|
original_name: s,
|
|
@@ -147,6 +245,15 @@ module Labkit
|
|
|
147
245
|
)
|
|
148
246
|
sanitized
|
|
149
247
|
end
|
|
248
|
+
|
|
249
|
+
# The name a rule runs under, without the warning, for comparisons that
|
|
250
|
+
# happen before prepare_rules logs it once.
|
|
251
|
+
def sanitized_name(name)
|
|
252
|
+
return name if RULE_NAME_PATTERN.match?(name) && name.length <= RULE_NAME_MAX_LENGTH
|
|
253
|
+
|
|
254
|
+
sanitized = name.downcase.gsub(/[^a-z0-9_]/, "_")[0, RULE_NAME_MAX_LENGTH]
|
|
255
|
+
sanitized.empty? ? "unnamed_rule" : sanitized
|
|
256
|
+
end
|
|
150
257
|
end
|
|
151
258
|
end
|
|
152
259
|
end
|
|
@@ -47,6 +47,12 @@ module Labkit
|
|
|
47
47
|
# presence matcher alongside the exclusions:
|
|
48
48
|
#
|
|
49
49
|
# path: [/./, { nre: '\A/api/' }]
|
|
50
|
+
#
|
|
51
|
+
# Rules from a config file are built with engine: :re2. Their :re and :nre
|
|
52
|
+
# patterns compile with RE2, the syntax every SDK shares, so a pattern that
|
|
53
|
+
# only Ruby accepts (lookbehind, a possessive quantifier) is rejected and a
|
|
54
|
+
# `^` anchors the whole text, not each line. RE2 has no backtracking, so
|
|
55
|
+
# those matchers need no timeout.
|
|
50
56
|
class Matcher < Data.define(:type, :value)
|
|
51
57
|
KNOWN_HASH_KEYS = %i[eq ne re nre oneOf noneOf].freeze
|
|
52
58
|
ERROR_INSPECT_LIMIT = 80
|
|
@@ -74,22 +80,27 @@ module Labkit
|
|
|
74
80
|
# nothing in practice, since callers pass Regexp constants.
|
|
75
81
|
MATCH_TIMEOUT_SECONDS = 0.05
|
|
76
82
|
|
|
77
|
-
|
|
83
|
+
ENGINES = %i[ruby re2].freeze
|
|
84
|
+
|
|
85
|
+
# engine is positional, not a keyword: callers write build(eq: x) and a
|
|
86
|
+
# keyword here would swallow that Hash.
|
|
87
|
+
def self.build(input, engine = :ruby)
|
|
88
|
+
raise ArgumentError, "engine must be one of #{ENGINES.inspect}, got #{engine.inspect}" unless ENGINES.include?(engine)
|
|
78
89
|
raise ArgumentError, nesting_error(input) if too_deeply_nested?(input)
|
|
79
90
|
|
|
80
91
|
case input
|
|
81
92
|
when Regexp
|
|
82
|
-
new(type: :re, value:
|
|
93
|
+
new(type: :re, value: regexp(:re, input, engine))
|
|
83
94
|
when Hash
|
|
84
|
-
from_hash(input)
|
|
95
|
+
from_hash(input, engine)
|
|
85
96
|
when Array
|
|
86
|
-
compose(input)
|
|
97
|
+
compose(input, engine)
|
|
87
98
|
else
|
|
88
99
|
new(type: :eq, value: input)
|
|
89
100
|
end
|
|
90
101
|
end
|
|
91
102
|
|
|
92
|
-
def self.from_hash(input)
|
|
103
|
+
def self.from_hash(input, engine)
|
|
93
104
|
raise ArgumentError, invalid_shape_error(input) if input.size != 1
|
|
94
105
|
|
|
95
106
|
type, source = input.first
|
|
@@ -109,11 +120,11 @@ module Labkit
|
|
|
109
120
|
"rate-limit match value has unknown type key #{truncate_for_error(type)}; accepted: #{KNOWN_HASH_KEYS.inspect}"
|
|
110
121
|
end
|
|
111
122
|
|
|
112
|
-
compile(type_sym, source)
|
|
123
|
+
compile(type_sym, source, engine)
|
|
113
124
|
end
|
|
114
125
|
private_class_method :from_hash
|
|
115
126
|
|
|
116
|
-
def self.compile(type_sym, source)
|
|
127
|
+
def self.compile(type_sym, source, engine)
|
|
117
128
|
case type_sym
|
|
118
129
|
when :eq, :ne
|
|
119
130
|
new(type: type_sym, value: source)
|
|
@@ -134,14 +145,19 @@ module Labkit
|
|
|
134
145
|
|
|
135
146
|
new(type: type_sym, value: Set.new(source).freeze)
|
|
136
147
|
when :re, :nre
|
|
137
|
-
new(type: type_sym, value:
|
|
148
|
+
new(type: type_sym, value: regexp(type_sym, source, engine))
|
|
138
149
|
end
|
|
139
150
|
end
|
|
140
151
|
private_class_method :compile
|
|
141
152
|
|
|
153
|
+
def self.regexp(key, source, engine)
|
|
154
|
+
engine == :re2 ? re2_regexp(key, source) : with_match_timeout(key, source)
|
|
155
|
+
end
|
|
156
|
+
private_class_method :regexp
|
|
157
|
+
|
|
142
158
|
# Builds the :all matcher behind the Array form. Entries go back through
|
|
143
159
|
# .build, which is what keeps a nested regex bounded by the match timeout.
|
|
144
|
-
def self.compose(input)
|
|
160
|
+
def self.compose(input, engine)
|
|
145
161
|
raise ArgumentError, "rate-limit match value [] must not be empty" if input.empty?
|
|
146
162
|
|
|
147
163
|
unless input.all? { |entry| entry.is_a?(Hash) || entry.is_a?(Regexp) }
|
|
@@ -150,10 +166,28 @@ module Labkit
|
|
|
150
166
|
"got #{truncate_for_error(input)}"
|
|
151
167
|
end
|
|
152
168
|
|
|
153
|
-
new(type: :all, value: input.map { |entry| build(entry) }.freeze)
|
|
169
|
+
new(type: :all, value: input.map { |entry| build(entry, engine) }.freeze)
|
|
154
170
|
end
|
|
155
171
|
private_class_method :compose
|
|
156
172
|
|
|
173
|
+
# The re2 gem is loaded here, not at require time, so a host that builds
|
|
174
|
+
# no file rules never pays for it.
|
|
175
|
+
# A Ruby Regexp with options cannot be handed to RE2 as its source alone:
|
|
176
|
+
# /api/i would silently turn case-sensitive.
|
|
177
|
+
def self.re2_regexp(key, source)
|
|
178
|
+
require "re2"
|
|
179
|
+
flagged = source.is_a?(Regexp) && source.options.anybits?(Regexp::IGNORECASE | Regexp::MULTILINE | Regexp::EXTENDED)
|
|
180
|
+
raise ArgumentError, "rate-limit match value {#{key}: #{truncate_for_error(source)}} has Regexp options, which RE2 does not take; write them into the pattern" if flagged
|
|
181
|
+
|
|
182
|
+
pattern = source.is_a?(Regexp) ? source.source : source.to_s
|
|
183
|
+
compiled = RE2::Regexp.new(pattern, log_errors: false)
|
|
184
|
+
return compiled if compiled.ok?
|
|
185
|
+
|
|
186
|
+
raise ArgumentError,
|
|
187
|
+
"rate-limit match value {#{key}: #{truncate_for_error(source)}} failed to compile: #{compiled.error}"
|
|
188
|
+
end
|
|
189
|
+
private_class_method :re2_regexp
|
|
190
|
+
|
|
157
191
|
# Compiles +source+ (a String pattern or an existing Regexp) into a
|
|
158
192
|
# Regexp bounded by MATCH_TIMEOUT_SECONDS.
|
|
159
193
|
#
|
|
@@ -41,11 +41,16 @@ module Labkit
|
|
|
41
41
|
# admitting rules after counter rules. Default false; nil
|
|
42
42
|
# means not set.
|
|
43
43
|
#
|
|
44
|
+
# source - :code for a rule built in Ruby (default), :file for a rule
|
|
45
|
+
# loaded from a rate limit config document. A :file rule
|
|
46
|
+
# compiles its patterns with RE2 and checks its name in
|
|
47
|
+
# every environment, because an operator wrote it.
|
|
48
|
+
#
|
|
44
49
|
# +name+ must be a lowercase alphanumeric-and-underscore string of at most 64
|
|
45
50
|
# characters. It is used as the middle segment of every Redis counter key for
|
|
46
51
|
# this rule, so changing a rule's name mid-window abandons its in-flight counters.
|
|
47
52
|
Rule = Data.define(
|
|
48
|
-
:name, :match, :limit, :period, :action, :characteristics, :count_distinct, :ban_for, :admit_counted_values
|
|
53
|
+
:name, :match, :limit, :period, :action, :characteristics, :count_distinct, :ban_for, :admit_counted_values, :source
|
|
49
54
|
) do
|
|
50
55
|
# ban_for changes the accounting; action still decides who gets blocked.
|
|
51
56
|
# A :skip rule counts nothing, so it has no limit to cross and no ban to
|
|
@@ -106,36 +111,47 @@ module Labkit
|
|
|
106
111
|
sym
|
|
107
112
|
end
|
|
108
113
|
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
)
|
|
114
|
+
# A :file rule is checked in every environment: an operator wrote it, and
|
|
115
|
+
# Limiter's production fallback of sanitising the name would silently
|
|
116
|
+
# change the counter key of a rule the operator can see in the file.
|
|
117
|
+
def self.validate_name!(name, source)
|
|
113
118
|
raise ArgumentError, "name must be a String or Symbol, got #{name.class}" unless name.is_a?(String) || name.is_a?(Symbol)
|
|
114
119
|
|
|
115
120
|
name_str = name.to_s
|
|
116
121
|
raise ArgumentError, "name must not be empty" if name_str.empty?
|
|
122
|
+
return name_str unless source == :file || Labkit.dev_or_test?
|
|
123
|
+
|
|
124
|
+
raise ArgumentError, "Invalid rule name: #{name.inspect}. Must match /\\A[a-z0-9_]+\\z/" unless RULE_NAME_PATTERN.match?(name_str)
|
|
125
|
+
raise ArgumentError, "Rule name too long: #{name.inspect}. Maximum 64 characters" if name_str.length > RULE_NAME_MAX_LENGTH
|
|
126
|
+
|
|
127
|
+
name_str
|
|
128
|
+
end
|
|
129
|
+
|
|
130
|
+
def initialize( # rubocop:disable Metrics/ParameterLists -- one keyword per Rule field; the public API grows by keywords with defaults
|
|
131
|
+
name:, limit:, period:, characteristics:,
|
|
132
|
+
match: {}, action: :limit, count_distinct: nil, ban_for: nil, admit_counted_values: false, source: :code
|
|
133
|
+
)
|
|
134
|
+
raise ArgumentError, "source must be :code or :file, got #{source.inspect}" unless %i[code file].include?(source)
|
|
135
|
+
|
|
136
|
+
name_str = self.class.validate_name!(name, source)
|
|
117
137
|
|
|
118
138
|
action_sym = action.to_sym
|
|
119
139
|
raise ArgumentError, "Invalid action: #{action.inspect}. Must be one of: #{KNOWN_ACTIONS.inspect}" unless KNOWN_ACTIONS.include?(action_sym)
|
|
120
140
|
|
|
121
|
-
if Labkit.dev_or_test?
|
|
122
|
-
raise ArgumentError, "Invalid rule name: #{name.inspect}. Must match /\\A[a-z0-9_]+\\z/" unless RULE_NAME_PATTERN.match?(name_str)
|
|
123
|
-
raise ArgumentError, "Rule name too long: #{name.inspect}. Maximum 64 characters" if name_str.length > RULE_NAME_MAX_LENGTH
|
|
124
|
-
end
|
|
125
|
-
|
|
126
141
|
characteristics_arr = Array(characteristics).map(&:to_sym).freeze
|
|
127
142
|
count_distinct_sym = self.class.normalize_count_distinct(count_distinct, characteristics_arr)
|
|
128
143
|
|
|
129
144
|
super(
|
|
130
145
|
name: name_str.freeze,
|
|
131
|
-
match: match.transform_keys(&:to_sym).transform_values { |v| Matcher.build(v) }.freeze,
|
|
146
|
+
match: match.transform_keys(&:to_sym).transform_values { |v| Matcher.build(v, source == :file ? :re2 : :ruby) }.freeze,
|
|
132
147
|
limit: self.class.normalize_numeric(:limit, limit, action_sym),
|
|
133
148
|
period: self.class.normalize_numeric(:period, period, action_sym),
|
|
134
149
|
action: action_sym,
|
|
135
150
|
characteristics: characteristics_arr,
|
|
136
151
|
count_distinct: count_distinct_sym,
|
|
137
152
|
ban_for: self.class.normalize_ban_for(ban_for, action_sym, count_distinct_sym),
|
|
138
|
-
admit_counted_values: self.class.normalize_admit_counted_values(admit_counted_values, action_sym, count_distinct_sym)
|
|
153
|
+
admit_counted_values: self.class.normalize_admit_counted_values(admit_counted_values, action_sym, count_distinct_sym),
|
|
154
|
+
source: source
|
|
139
155
|
)
|
|
140
156
|
end
|
|
141
157
|
end
|
data/lib/labkit/rate_limit.rb
CHANGED
|
@@ -51,7 +51,9 @@ module Labkit
|
|
|
51
51
|
# callables on rule limit/period; see Limiter#check
|
|
52
52
|
# @return [Result]
|
|
53
53
|
def check(name:, identifier:, rules:, redis: nil, logger: nil, cost: 1, rule_context: nil)
|
|
54
|
-
|
|
54
|
+
# A throw-away limiter runs the rules it is given; the configured rule
|
|
55
|
+
# document applies to limiters a host builds and keeps.
|
|
56
|
+
Limiter.new(name: name, rules: rules, redis: redis, logger: logger, document: nil)
|
|
55
57
|
.check(identifier, cost: cost, rule_context: rule_context)
|
|
56
58
|
end
|
|
57
59
|
end
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: gitlab-labkit
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 5.
|
|
4
|
+
version: 5.13.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Andrew Newdigate
|
|
@@ -257,6 +257,26 @@ dependencies:
|
|
|
257
257
|
- - "<"
|
|
258
258
|
- !ruby/object:Gem::Version
|
|
259
259
|
version: '2.0'
|
|
260
|
+
- !ruby/object:Gem::Dependency
|
|
261
|
+
name: re2
|
|
262
|
+
requirement: !ruby/object:Gem::Requirement
|
|
263
|
+
requirements:
|
|
264
|
+
- - ">="
|
|
265
|
+
- !ruby/object:Gem::Version
|
|
266
|
+
version: '2.15'
|
|
267
|
+
- - "<"
|
|
268
|
+
- !ruby/object:Gem::Version
|
|
269
|
+
version: '3'
|
|
270
|
+
type: :runtime
|
|
271
|
+
prerelease: false
|
|
272
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
273
|
+
requirements:
|
|
274
|
+
- - ">="
|
|
275
|
+
- !ruby/object:Gem::Version
|
|
276
|
+
version: '2.15'
|
|
277
|
+
- - "<"
|
|
278
|
+
- !ruby/object:Gem::Version
|
|
279
|
+
version: '3'
|
|
260
280
|
- !ruby/object:Gem::Dependency
|
|
261
281
|
name: redis
|
|
262
282
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -652,7 +672,9 @@ files:
|
|
|
652
672
|
- lib/labkit/rate_limit/README.md
|
|
653
673
|
- lib/labkit/rate_limit/block_summary.rb
|
|
654
674
|
- lib/labkit/rate_limit/config.rb
|
|
675
|
+
- lib/labkit/rate_limit/config/document.rb
|
|
655
676
|
- lib/labkit/rate_limit/config/error.rb
|
|
677
|
+
- lib/labkit/rate_limit/config/loader.rb
|
|
656
678
|
- lib/labkit/rate_limit/config/parser.rb
|
|
657
679
|
- lib/labkit/rate_limit/config/proto/config_pb.rb
|
|
658
680
|
- lib/labkit/rate_limit/config/schema.rb
|