gitlab-labkit 5.10.0 → 5.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/gitlab-labkit.gemspec +2 -1
- data/lib/labkit/rate_limit/README.md +15 -4
- data/lib/labkit/rate_limit/config/parser.rb +1 -2
- data/lib/labkit/rate_limit/config/schema.rb +324 -0
- data/lib/labkit/rate_limit/config.rb +1 -0
- data/scripts/vendor-ratelimit-proto.sh +21 -10
- metadata +2 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: adf3a44ad840a0b124ef408e68697340e1c62ca66e0965da67ed3e21aa608a92
|
|
4
|
+
data.tar.gz: d3ccd488458bc4c365c476217e6b1234bb7abec41efee94413d81b69497ff460
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 7bbbd90bd5eb0e65f794845225fd8707817bea9d8a80f47da64063b4e8e9a7c988177dae2ad0f44e0fb7ca22dca6dd2846e0fc2f2dd9db038c4554d2738d4364
|
|
7
|
+
data.tar.gz: c15ff6de4430a52692e33dd685ca88e4e080a12c6026f4c59f6520b91171113db5e9d13e319d1ac783e9dfea2b5648637ab4c0c8224feedc927ef2f70bc704ea
|
data/gitlab-labkit.gemspec
CHANGED
|
@@ -15,7 +15,8 @@ Gem::Specification.new do |spec|
|
|
|
15
15
|
spec.metadata = { "source_code_uri" => "https://gitlab.com/gitlab-org/ruby/gems/labkit-ruby" }
|
|
16
16
|
spec.license = "MIT"
|
|
17
17
|
|
|
18
|
-
|
|
18
|
+
# The vendored .proto sources are read by the specs only; the gem ships the generated Ruby.
|
|
19
|
+
spec.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(spec|tools)/}) || f.end_with?(".proto") }
|
|
19
20
|
spec.bindir = "exe"
|
|
20
21
|
spec.executables = %w[labkit-logging]
|
|
21
22
|
spec.require_paths = ["lib"]
|
|
@@ -724,19 +724,30 @@ A gem schema that lacks or changes a vendored field raises `Labkit::RateLimit::C
|
|
|
724
724
|
So upgrade the `gitlab-ratelimit-config-v1` gem in the application first, then the `gitlab-labkit` release that updates the vendored copy.
|
|
725
725
|
|
|
726
726
|
Loading runs in four steps, `parse`, `schema`, `contract` and `rule`, and a failure raises `Labkit::RateLimit::Config::Error` whose `#step` names the step.
|
|
727
|
-
|
|
728
|
-
Until the schema step lands, no mode validates a document: strict mode lets an enum number that no `Action` defines through, such as `action: 7`, and lenient mode leaves an unknown `action` name as `ACTION_UNSPECIFIED`.
|
|
729
|
-
The schema step rejects both.
|
|
727
|
+
The parse and schema steps exist; the contract and rule steps follow in [Part 3 of the plan](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/30263).
|
|
730
728
|
|
|
731
729
|
```ruby
|
|
732
730
|
doc = Labkit::RateLimit::Config::Parser.parse(
|
|
733
731
|
"config/rate_limits/rack_request.yaml",
|
|
734
732
|
Labkit::RateLimit::Config::Proto::RateLimits,
|
|
735
|
-
strict: true # CI rejects unknown fields. Production loads with strict: false
|
|
733
|
+
strict: true # CI rejects unknown fields. Production loads with strict: false
|
|
736
734
|
)
|
|
735
|
+
Labkit::RateLimit::Config::Schema.check!(doc)
|
|
737
736
|
doc.rate_limits["rack_request"].rules.first.name
|
|
738
737
|
```
|
|
739
738
|
|
|
739
|
+
`Schema.check!` mirrors every protovalidate rule of the `gitlab.ratelimit.config.v1` schema: name format, the period presets, the `limit` and `ban_for_s` ranges, every list cap, and the message rules such as `rule.limit_requires_threshold`.
|
|
740
|
+
It accepts `AvailableLimiters`, `RateLimits` and `PublishRulesRequest`.
|
|
741
|
+
A failure lists every violation with the field path, message and rule id that protovalidate reports, such as `rate_limits["rack_request"].rules[0].period_s: must be in list [1, 15, 30, 60, 180, 300, 600, 900, 1800, 3600, 28800, 86400] (uint32.in)`, so a failure reads the same in Ruby and Go.
|
|
742
|
+
`Schema.violations(doc)` returns them without raising.
|
|
743
|
+
Inside one message the order is protovalidate's: message rules, then the oneof, then fields in declaration order.
|
|
744
|
+
A protobuf map keeps no document order and Go iterates it in an unspecified order, so Ruby reports map entries by sorted key and a cross-SDK comparison matches violations per entry, not by position.
|
|
745
|
+
Map keys in a path are quoted as Go's `strconv.Quote` prints them.
|
|
746
|
+
The check is the strict reading of the schema: a matcher whose operator this binary does not know fails `required`.
|
|
747
|
+
Lenient loading, through the loader that follows in Part 3, drops such a rule with a warning before the schema step, so a newer operator does not reject the file.
|
|
748
|
+
`scripts/vendor-ratelimit-proto.sh` also copies `config.proto` and `constraint.proto` next to the generated code, and `schema_spec.rb` parses them and compares every field rule (caps, bounds, presets, flags), every message rule id and message, and the name pattern with `Schema`, so a `PROTOS_REF` bump that changes any of those fails the suite until `Schema` follows.
|
|
749
|
+
The CEL expressions themselves are mirrored by hand and pinned by the examples.
|
|
750
|
+
|
|
740
751
|
The parse step converts each YAML scalar by the type of the field it fills, so `eq: off`, `eq: True` and `eq: 9970` stay strings, `limit: "5"` becomes `5`, `action: "1"` becomes `ACTION_LIMIT`, and a YAML null leaves the field unset.
|
|
741
752
|
A repeated key, also one repeated under its camelCase `json_name`, a merge key (`<<`), an empty file, more than one document, or a non-scalar key fails at step `parse`, in both modes.
|
|
742
753
|
So does a value of the wrong kind for its field, such as a mapping where a scalar belongs, an integer out of range for its field, two fields of one `oneof`, such as `{ eq: a, re: b }`, a null list item or map value, and a document nested deeper than 64 levels.
|
|
@@ -22,8 +22,7 @@ module Labkit
|
|
|
22
22
|
# number that no Action defines, such as `action: 7`, passes in both
|
|
23
23
|
# modes, and in lenient mode protobuf drops an unknown enum name together
|
|
24
24
|
# with the unknown fields, so a rule with a misspelt action reaches the
|
|
25
|
-
# schema step as ACTION_UNSPECIFIED.
|
|
26
|
-
# both.
|
|
25
|
+
# schema step as ACTION_UNSPECIFIED. Schema.check! rejects both.
|
|
27
26
|
module Parser
|
|
28
27
|
INT32 = (-(2**31))...(2**31)
|
|
29
28
|
UINT32 = 0...(2**32)
|
|
@@ -0,0 +1,324 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Labkit
|
|
4
|
+
module RateLimit
|
|
5
|
+
module Config
|
|
6
|
+
# Schema is the schema step: the checks that gitlab/ratelimit/config/v1
|
|
7
|
+
# declares with protovalidate, mirrored in Ruby because there is no
|
|
8
|
+
# protovalidate runtime for Ruby. Each check keeps the protovalidate rule
|
|
9
|
+
# id, message and field path. Inside one message the order is
|
|
10
|
+
# protovalidate's: message rules, then the oneof, then fields in
|
|
11
|
+
# declaration order.
|
|
12
|
+
# A protobuf map keeps no document order and Go iterates it in an
|
|
13
|
+
# unspecified order, so Ruby reports map entries by sorted key and the
|
|
14
|
+
# corpus compares per entry.
|
|
15
|
+
#
|
|
16
|
+
# FIELD_RULES and MESSAGE_RULES are copied from the proto by hand, and
|
|
17
|
+
# schema_spec.rb compares both with the vendored config.proto and
|
|
18
|
+
# constraint.proto field by field, so a PROTOS_REF bump that changes a
|
|
19
|
+
# cap, a flag, a rule id or a message fails the suite until this file
|
|
20
|
+
# follows. The CEL expressions have no such check; the examples pin them.
|
|
21
|
+
#
|
|
22
|
+
# The check is the strict reading of the schema. Lenient loading drops a
|
|
23
|
+
# rule whose matcher has no known operator before this step (the loader,
|
|
24
|
+
# Task 3.3), because such a matcher fails `required` here.
|
|
25
|
+
module Schema
|
|
26
|
+
# for_key marks a violation on a map key; its path is the entry's path.
|
|
27
|
+
Violation = Data.define(:path, :rule_id, :message, :for_key) do
|
|
28
|
+
def initialize(path:, rule_id:, message:, for_key: false)
|
|
29
|
+
super
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
def to_s
|
|
33
|
+
path.empty? ? "#{message} (#{rule_id})" : "#{path}: #{message} (#{rule_id})"
|
|
34
|
+
end
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
# CEL matches() is RE2, where ^ and $ anchor the whole text; Ruby's $
|
|
38
|
+
# matches before a newline, so the mirror anchors with \A and \z.
|
|
39
|
+
NAME = /\A[a-z0-9_]{1,64}\z/
|
|
40
|
+
NAME_MESSAGE = "invalid name; must match ^[a-z0-9_]{1,64}$"
|
|
41
|
+
PERIODS = [1, 15, 30, 60, 180, 300, 600, 900, 1800, 3600, 28800, 86400].freeze
|
|
42
|
+
SKIP = :ACTION_SKIP
|
|
43
|
+
|
|
44
|
+
# The field rules of config.proto, keyed "Message.field". The keys of a
|
|
45
|
+
# map field and the items of a repeated field carry their rule under
|
|
46
|
+
# keys_* and items_*.
|
|
47
|
+
FIELD_RULES = {
|
|
48
|
+
"AvailableLimiters.available_limiters" => { min_pairs: 1, keys_is_name: true },
|
|
49
|
+
"LimiterSpec.description" => { max_len: 1024 },
|
|
50
|
+
"LimiterSpec.available_properties" => { unique: true, max_items: 100, items_is_name: true },
|
|
51
|
+
"LimiterSpec.default_rules" => { max_items: 200 },
|
|
52
|
+
"LimiterSpec.protected_rules" => { unique: true, max_items: 200, items_is_name: true },
|
|
53
|
+
"RateLimits.rate_limits" => { keys_is_name: true },
|
|
54
|
+
"RuleList.rules" => { max_items: 200 },
|
|
55
|
+
"Rule.name" => { is_name: true },
|
|
56
|
+
"Rule.description" => { max_len: 1024 },
|
|
57
|
+
"Rule.action" => { required: true, defined_only: true },
|
|
58
|
+
"Rule.characteristics" => { unique: true, max_items: 10, items_is_name: true },
|
|
59
|
+
"Rule.match" => { max_pairs: 20, keys_is_name: true },
|
|
60
|
+
"Rule.limit" => { gte: 1, lte: 1_000_000 },
|
|
61
|
+
"Rule.period_s" => { in: PERIODS },
|
|
62
|
+
"Rule.count_distinct" => { is_name: true, ignore_empty: true },
|
|
63
|
+
"Rule.ban_for_s" => { gte: 1, lte: 86_400 },
|
|
64
|
+
"PublishRulesRequest.rate_limits" => { keys_is_name: true },
|
|
65
|
+
"PublishRulesRequest.remove" => { keys_is_name: true },
|
|
66
|
+
"RuleNames.names" => { min_items: 1, max_items: 200, unique: true, items_is_name: true },
|
|
67
|
+
"Matcher.eq" => { max_len: 1024 },
|
|
68
|
+
"Matcher.ne" => { max_len: 1024 },
|
|
69
|
+
"Matcher.re" => { max_len: 1024 },
|
|
70
|
+
"Matcher.nre" => { max_len: 1024 },
|
|
71
|
+
"StringList.values" => { min_items: 1, max_items: 100, unique: true, items_max_len: 1024 },
|
|
72
|
+
"MatcherList.matchers" => { min_items: 1, max_items: 20 }
|
|
73
|
+
}.freeze
|
|
74
|
+
|
|
75
|
+
# The message-level CEL rules of config.proto, by id.
|
|
76
|
+
MESSAGE_RULES = {
|
|
77
|
+
"limiter_spec.protected_rules_declared" => "every protected rule must be a default rule of this limiter",
|
|
78
|
+
"limiter_spec.unique_default_names" => "default rule names must be unique within a limiter",
|
|
79
|
+
"limiter_spec.no_position_in_defaults" => "position is not allowed on a default rule",
|
|
80
|
+
"rule_list.unique_names" => "rule names must be unique within a limiter",
|
|
81
|
+
"rule.limit_requires_threshold" => "limit and period_s are required unless action is skip",
|
|
82
|
+
"rule.ban_excludes_count_distinct" => "ban_for_s cannot be combined with count_distinct",
|
|
83
|
+
"rule.ban_not_on_skip" => "ban_for_s is not allowed on a skip rule",
|
|
84
|
+
"rule.no_counter_on_skip" => "limit, period_s, characteristics and count_distinct are not allowed on a skip rule",
|
|
85
|
+
"rule.count_distinct_not_characteristic" => "count_distinct cannot also be a characteristic",
|
|
86
|
+
"publish.name_in_both" => "a rule name cannot be in both rate_limits and remove for one limiter",
|
|
87
|
+
"matcher_list.no_nested_all" => "all cannot contain another all"
|
|
88
|
+
}.freeze
|
|
89
|
+
|
|
90
|
+
# The ASCII characters that strconv.Quote prints with a backslash escape.
|
|
91
|
+
ESCAPES = {
|
|
92
|
+
'"' => '\\"', "\\" => "\\\\", "\a" => "\\a", "\b" => "\\b", "\f" => "\\f",
|
|
93
|
+
"\n" => "\\n", "\r" => "\\r", "\t" => "\\t", "\v" => "\\v"
|
|
94
|
+
}.freeze
|
|
95
|
+
|
|
96
|
+
class << self
|
|
97
|
+
# @param message [Proto::AvailableLimiters, Proto::RateLimits, Proto::PublishRulesRequest]
|
|
98
|
+
# @return the message
|
|
99
|
+
# @raise [Error] step :schema, every violation joined with "; "
|
|
100
|
+
def check!(message)
|
|
101
|
+
found = violations(message)
|
|
102
|
+
raise Error.new(:schema, found.join("; ")) unless found.empty?
|
|
103
|
+
|
|
104
|
+
message
|
|
105
|
+
end
|
|
106
|
+
|
|
107
|
+
# @return [Array<Violation>] every violation, in the order described above
|
|
108
|
+
def violations(message)
|
|
109
|
+
checker = Checker.new
|
|
110
|
+
case message
|
|
111
|
+
when Proto::AvailableLimiters then checker.available_limiters(message)
|
|
112
|
+
when Proto::RateLimits then checker.rate_limits(message)
|
|
113
|
+
when Proto::PublishRulesRequest then checker.publish_rules_request(message)
|
|
114
|
+
else
|
|
115
|
+
raise ArgumentError, "no schema for #{message.class}; expected AvailableLimiters, RateLimits or PublishRulesRequest"
|
|
116
|
+
end
|
|
117
|
+
checker.violations
|
|
118
|
+
end
|
|
119
|
+
|
|
120
|
+
# Quotes a map key for a path the way Go's strconv.Quote does, so the
|
|
121
|
+
# path reads the same in every SDK: backslash escapes for the ASCII
|
|
122
|
+
# controls that have one, \xNN for the other ASCII controls and DEL,
|
|
123
|
+
# \uNNNN or \UNNNNNNNN for a non-printable rune, everything else as is.
|
|
124
|
+
def quote(key)
|
|
125
|
+
out = +'"'
|
|
126
|
+
key.each_char { |c| out << (ESCAPES[c] || escape_rune(c)) }
|
|
127
|
+
out << '"'
|
|
128
|
+
end
|
|
129
|
+
|
|
130
|
+
private
|
|
131
|
+
|
|
132
|
+
# Go's unicode.IsPrint: letters, marks, numbers, punctuation, symbols
|
|
133
|
+
# and the ASCII space.
|
|
134
|
+
def escape_rune(char)
|
|
135
|
+
code = char.ord
|
|
136
|
+
return format("\\x%02x", code) if code < 0x20 || code == 0x7f
|
|
137
|
+
return char if char == " " || char.match?(/\A[\p{L}\p{M}\p{N}\p{P}\p{S}]\z/)
|
|
138
|
+
|
|
139
|
+
code > 0xffff ? format("\\U%08x", code) : format("\\u%04x", code)
|
|
140
|
+
end
|
|
141
|
+
end
|
|
142
|
+
|
|
143
|
+
# Walks one message and collects violations. Paths follow protovalidate:
|
|
144
|
+
# `rate_limits["rack_request"].rules[0].match["plan"].eq`.
|
|
145
|
+
class Checker
|
|
146
|
+
attr_reader :violations
|
|
147
|
+
|
|
148
|
+
def initialize
|
|
149
|
+
@violations = []
|
|
150
|
+
end
|
|
151
|
+
|
|
152
|
+
def available_limiters(doc)
|
|
153
|
+
map_field("available_limiters", doc.available_limiters, FIELD_RULES["AvailableLimiters.available_limiters"]) { |path, spec| limiter_spec(path, spec) }
|
|
154
|
+
end
|
|
155
|
+
|
|
156
|
+
def rate_limits(doc)
|
|
157
|
+
map_field("rate_limits", doc.rate_limits, FIELD_RULES["RateLimits.rate_limits"]) { |path, list| rule_list(path, list) }
|
|
158
|
+
end
|
|
159
|
+
|
|
160
|
+
def publish_rules_request(req)
|
|
161
|
+
# The outcome of one name in both fields would depend on the order
|
|
162
|
+
# the store applies them.
|
|
163
|
+
both = req.rate_limits.any? do |key, list|
|
|
164
|
+
req.remove.has_key?(key) && list.rules.any? { |r| req.remove[key].names.include?(r.name) }
|
|
165
|
+
end
|
|
166
|
+
message_rule("", "publish.name_in_both") if both
|
|
167
|
+
map_field("rate_limits", req.rate_limits, FIELD_RULES["PublishRulesRequest.rate_limits"]) { |path, list| rule_list(path, list) }
|
|
168
|
+
map_field("remove", req.remove, FIELD_RULES["PublishRulesRequest.remove"]) do |path, names|
|
|
169
|
+
list("#{path}.names", names.names, FIELD_RULES["RuleNames.names"])
|
|
170
|
+
end
|
|
171
|
+
end
|
|
172
|
+
|
|
173
|
+
private
|
|
174
|
+
|
|
175
|
+
def limiter_spec(path, spec)
|
|
176
|
+
default_names = spec.default_rules.map(&:name)
|
|
177
|
+
message_rule(path, "limiter_spec.protected_rules_declared") unless spec.protected_rules.all? { |n| default_names.include?(n) }
|
|
178
|
+
message_rule(path, "limiter_spec.unique_default_names") unless default_names.uniq.size == default_names.size
|
|
179
|
+
# Default rules keep document order, so a position has no meaning there.
|
|
180
|
+
message_rule(path, "limiter_spec.no_position_in_defaults") if spec.default_rules.any?(&:has_position?)
|
|
181
|
+
|
|
182
|
+
string("#{path}.description", spec.description, FIELD_RULES["LimiterSpec.description"])
|
|
183
|
+
list("#{path}.available_properties", spec.available_properties, FIELD_RULES["LimiterSpec.available_properties"])
|
|
184
|
+
list("#{path}.default_rules", spec.default_rules, FIELD_RULES["LimiterSpec.default_rules"])
|
|
185
|
+
spec.default_rules.each_with_index { |r, i| rule("#{path}.default_rules[#{i}]", r) }
|
|
186
|
+
list("#{path}.protected_rules", spec.protected_rules, FIELD_RULES["LimiterSpec.protected_rules"])
|
|
187
|
+
end
|
|
188
|
+
|
|
189
|
+
def rule_list(path, list)
|
|
190
|
+
names = list.rules.map(&:name)
|
|
191
|
+
message_rule(path, "rule_list.unique_names") unless names.uniq.size == names.size
|
|
192
|
+
list("#{path}.rules", list.rules, FIELD_RULES["RuleList.rules"])
|
|
193
|
+
list.rules.each_with_index { |r, i| rule("#{path}.rules[#{i}]", r) }
|
|
194
|
+
end
|
|
195
|
+
|
|
196
|
+
def rule(path, rule)
|
|
197
|
+
rule_message_rules(path, rule)
|
|
198
|
+
string("#{path}.name", rule.name, FIELD_RULES["Rule.name"])
|
|
199
|
+
string("#{path}.description", rule.description, FIELD_RULES["Rule.description"])
|
|
200
|
+
action("#{path}.action", rule.action, FIELD_RULES["Rule.action"])
|
|
201
|
+
list("#{path}.characteristics", rule.characteristics, FIELD_RULES["Rule.characteristics"])
|
|
202
|
+
map_field("#{path}.match", rule.match, FIELD_RULES["Rule.match"]) { |entry, matcher| matcher(entry, matcher) }
|
|
203
|
+
integer("#{path}.limit", rule.limit, FIELD_RULES["Rule.limit"]) if rule.has_limit?
|
|
204
|
+
integer("#{path}.period_s", rule.period_s, FIELD_RULES["Rule.period_s"]) if rule.has_period_s?
|
|
205
|
+
string("#{path}.count_distinct", rule.count_distinct, FIELD_RULES["Rule.count_distinct"])
|
|
206
|
+
integer("#{path}.ban_for_s", rule.ban_for_s, FIELD_RULES["Rule.ban_for_s"]) if rule.has_ban_for_s?
|
|
207
|
+
end
|
|
208
|
+
|
|
209
|
+
# In the order config.proto declares them.
|
|
210
|
+
def rule_message_rules(path, rule)
|
|
211
|
+
counts_distinct = !rule.count_distinct.empty?
|
|
212
|
+
message_rule(path, "rule.limit_requires_threshold") unless rule.action == SKIP || (rule.has_limit? && rule.has_period_s?)
|
|
213
|
+
message_rule(path, "rule.ban_excludes_count_distinct") if rule.has_ban_for_s? && counts_distinct
|
|
214
|
+
skip_rules(path, rule) if rule.action == SKIP
|
|
215
|
+
message_rule(path, "rule.count_distinct_not_characteristic") if counts_distinct && rule.characteristics.include?(rule.count_distinct)
|
|
216
|
+
end
|
|
217
|
+
|
|
218
|
+
# A skip rule has no counter and no limit to cross.
|
|
219
|
+
def skip_rules(path, rule)
|
|
220
|
+
counter = rule.has_limit? || rule.has_period_s? || !rule.characteristics.empty? || !rule.count_distinct.empty?
|
|
221
|
+
message_rule(path, "rule.ban_not_on_skip") if rule.has_ban_for_s?
|
|
222
|
+
message_rule(path, "rule.no_counter_on_skip") if counter
|
|
223
|
+
end
|
|
224
|
+
|
|
225
|
+
# An unknown enum number decodes as an Integer in Ruby.
|
|
226
|
+
def action(path, value, rules)
|
|
227
|
+
if rules[:required] && value == :ACTION_UNSPECIFIED
|
|
228
|
+
add(path, "required", "value is required")
|
|
229
|
+
elsif rules[:defined_only] && value.is_a?(Integer)
|
|
230
|
+
add(path, "enum.defined_only", "value must be one of the defined enum values")
|
|
231
|
+
end
|
|
232
|
+
end
|
|
233
|
+
|
|
234
|
+
def matcher(path, matcher)
|
|
235
|
+
op = matcher.op
|
|
236
|
+
return add("#{path}.op", "required", "exactly one field is required in oneof") if op.nil?
|
|
237
|
+
|
|
238
|
+
case op
|
|
239
|
+
when :eq, :ne, :re, :nre then string("#{path}.#{op}", matcher[op.to_s], FIELD_RULES["Matcher.#{op}"])
|
|
240
|
+
when :one_of, :none_of then list("#{path}.#{op}.values", matcher[op.to_s].values, FIELD_RULES["StringList.values"])
|
|
241
|
+
when :all then matcher_list("#{path}.all", matcher.all)
|
|
242
|
+
end
|
|
243
|
+
end
|
|
244
|
+
|
|
245
|
+
def matcher_list(path, list)
|
|
246
|
+
# There is no `any`, so an `all` inside an `all` is one flat list.
|
|
247
|
+
message_rule(path, "matcher_list.no_nested_all") if list.matchers.any? { |m| m.op == :all }
|
|
248
|
+
list("#{path}.matchers", list.matchers, FIELD_RULES["MatcherList.matchers"])
|
|
249
|
+
list.matchers.each_with_index { |m, i| matcher("#{path}.matchers[#{i}]", m) }
|
|
250
|
+
end
|
|
251
|
+
|
|
252
|
+
# Rules of RepeatedRules in field order: min_items, max_items, unique,
|
|
253
|
+
# then the items.
|
|
254
|
+
def list(path, values, rules)
|
|
255
|
+
add(path, "repeated.min_items", "must contain at least #{rules[:min_items]} item(s)") if rules[:min_items] && values.size < rules[:min_items]
|
|
256
|
+
add(path, "repeated.max_items", "must contain no more than #{rules[:max_items]} item(s)") if rules[:max_items] && values.size > rules[:max_items]
|
|
257
|
+
add(path, "repeated.unique", "repeated value must contain unique items") if rules[:unique] && values.uniq.size != values.size
|
|
258
|
+
values.each_with_index { |v, i| name("#{path}[#{i}]", v) } if rules[:items_is_name]
|
|
259
|
+
values.each_with_index { |v, i| max_len("#{path}[#{i}]", v, rules[:items_max_len]) } if rules[:items_max_len]
|
|
260
|
+
end
|
|
261
|
+
|
|
262
|
+
# Rules of MapRules in field order: min_pairs, max_pairs, then the
|
|
263
|
+
# keys. Yields each entry's path and value.
|
|
264
|
+
def map_field(path, map, rules)
|
|
265
|
+
add(path, "map.min_pairs", "map must be at least #{rules[:min_pairs]} entries") if rules[:min_pairs] && map.length < rules[:min_pairs]
|
|
266
|
+
add(path, "map.max_pairs", "map must be at most #{rules[:max_pairs]} entries") if rules[:max_pairs] && map.length > rules[:max_pairs]
|
|
267
|
+
map.keys.sort.each do |key|
|
|
268
|
+
value = map[key]
|
|
269
|
+
entry = "#{path}[#{Schema.quote(key)}]"
|
|
270
|
+
add(entry, "string.is_name", NAME_MESSAGE, for_key: true) if rules[:keys_is_name] && !NAME.match?(key)
|
|
271
|
+
yield entry, value
|
|
272
|
+
end
|
|
273
|
+
end
|
|
274
|
+
|
|
275
|
+
def string(path, value, rules)
|
|
276
|
+
return if rules[:ignore_empty] && value.empty?
|
|
277
|
+
|
|
278
|
+
max_len(path, value, rules[:max_len]) if rules[:max_len]
|
|
279
|
+
name(path, value) if rules[:is_name]
|
|
280
|
+
end
|
|
281
|
+
|
|
282
|
+
# protovalidate reports one of three ids for bounds, by which bounds
|
|
283
|
+
# the field declares.
|
|
284
|
+
def integer(path, value, rules)
|
|
285
|
+
if rules[:in]
|
|
286
|
+
add(path, "uint32.in", "must be in list #{rules[:in]}") unless rules[:in].include?(value)
|
|
287
|
+
return
|
|
288
|
+
end
|
|
289
|
+
|
|
290
|
+
gte, lte = rules.values_at(:gte, :lte)
|
|
291
|
+
below = gte && value < gte
|
|
292
|
+
above = lte && value > lte
|
|
293
|
+
return unless below || above
|
|
294
|
+
|
|
295
|
+
if gte && lte
|
|
296
|
+
add(path, "uint32.gte_lte", "must be greater than or equal to #{gte} and less than or equal to #{lte}")
|
|
297
|
+
elsif below
|
|
298
|
+
add(path, "uint32.gte", "must be greater than or equal to #{gte}")
|
|
299
|
+
else
|
|
300
|
+
add(path, "uint32.lte", "must be less than or equal to #{lte}")
|
|
301
|
+
end
|
|
302
|
+
end
|
|
303
|
+
|
|
304
|
+
def message_rule(path, rule_id)
|
|
305
|
+
add(path, rule_id, MESSAGE_RULES.fetch(rule_id))
|
|
306
|
+
end
|
|
307
|
+
|
|
308
|
+
def name(path, value)
|
|
309
|
+
add(path, "string.is_name", NAME_MESSAGE) unless NAME.match?(value)
|
|
310
|
+
end
|
|
311
|
+
|
|
312
|
+
def max_len(path, value, max)
|
|
313
|
+
add(path, "string.max_len", "must be at most #{max} characters") if value.length > max
|
|
314
|
+
end
|
|
315
|
+
|
|
316
|
+
def add(path, rule_id, message, for_key: false)
|
|
317
|
+
@violations << Violation.new(path: path, rule_id: rule_id, message: message, for_key: for_key)
|
|
318
|
+
end
|
|
319
|
+
end
|
|
320
|
+
private_constant :Checker
|
|
321
|
+
end
|
|
322
|
+
end
|
|
323
|
+
end
|
|
324
|
+
end
|
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
#!/usr/bin/env bash
|
|
2
|
-
# Copies the generated Ruby code for gitlab.ratelimit.config.v1
|
|
2
|
+
# Copies the generated Ruby code and the proto sources for gitlab.ratelimit.config.v1
|
|
3
|
+
# from gitlab-org/protos. The sources are what schema_spec.rb checks Schema against.
|
|
3
4
|
# A published gem cannot depend on the protos Bundler git gem, so the file is
|
|
4
5
|
# vendored and CI checks that it matches PROTOS_REF (job vendored-proto-check).
|
|
5
6
|
set -euo pipefail
|
|
@@ -7,12 +8,20 @@ set -euo pipefail
|
|
|
7
8
|
# Pinned protos module tag. Bump it when a newer schema release is needed.
|
|
8
9
|
PROTOS_REF="${PROTOS_REF:-gen/ruby/gitlab/ratelimit/config/v1/v0.1.0}"
|
|
9
10
|
SOURCE_PATH="gen/ruby/gitlab/ratelimit/config/v1/lib/gitlab/ratelimit/config/v1/config_pb.rb"
|
|
10
|
-
|
|
11
|
+
PROTO_DIR="proto/ratelimit/gitlab/ratelimit/config/v1"
|
|
12
|
+
TARGET_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/lib/labkit/rate_limit/config/proto"
|
|
13
|
+
TARGET="${TARGET_DIR}/config_pb.rb"
|
|
11
14
|
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
15
|
+
raw="https://gitlab.com/gitlab-org/protos/-/raw/${PROTOS_REF}"
|
|
16
|
+
# Everything lands in a temp dir first, so a failed download or check
|
|
17
|
+
# leaves the vendored files as they were.
|
|
18
|
+
work="$(mktemp -d)"
|
|
19
|
+
trap 'rm -rf "$work"' EXIT
|
|
20
|
+
tmp="${work}/config_pb.rb"
|
|
21
|
+
curl --fail --silent --show-error --location "${raw}/${SOURCE_PATH}" -o "$tmp"
|
|
22
|
+
for proto in config.proto constraint.proto; do
|
|
23
|
+
curl --fail --silent --show-error --location "${raw}/${PROTO_DIR}/${proto}" -o "${work}/${proto}"
|
|
24
|
+
done
|
|
16
25
|
|
|
17
26
|
# The generated file registers gitlab/ratelimit/config/v1/config.proto in the
|
|
18
27
|
# global descriptor pool. The protos gem registers the same file name, so the
|
|
@@ -27,10 +36,12 @@ guard='return unless Labkit::RateLimit::Config::VendoredProto.register(pool, des
|
|
|
27
36
|
grep -v '^# frozen_string_literal: true$' "$tmp" |
|
|
28
37
|
sed -e "s|^require 'google/protobuf'\$|require 'google/protobuf'\nrequire 'labkit/rate_limit/config/vendored_proto'|" \
|
|
29
38
|
-e "s|^pool.add_serialized_file(descriptor_data)\$|${guard}|"
|
|
30
|
-
} >"$
|
|
31
|
-
if ! grep -q 'VendoredProto.register' "$
|
|
32
|
-
! grep -qx "require 'labkit/rate_limit/config/vendored_proto'" "$
|
|
39
|
+
} >"${work}/vendored.rb"
|
|
40
|
+
if ! grep -q 'VendoredProto.register' "${work}/vendored.rb" ||
|
|
41
|
+
! grep -qx "require 'labkit/rate_limit/config/vendored_proto'" "${work}/vendored.rb"; then
|
|
33
42
|
echo "vendor: the generated file no longer requires VendoredProto or registers the descriptor as expected; update the sed in this script" >&2
|
|
34
43
|
exit 1
|
|
35
44
|
fi
|
|
36
|
-
|
|
45
|
+
mv "${work}/vendored.rb" "$TARGET"
|
|
46
|
+
mv "${work}/config.proto" "${work}/constraint.proto" "$TARGET_DIR/"
|
|
47
|
+
echo "vendored ${SOURCE_PATH}, config.proto and constraint.proto @${PROTOS_REF} -> ${TARGET_DIR#"$PWD"/}"
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: gitlab-labkit
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 5.
|
|
4
|
+
version: 5.11.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Andrew Newdigate
|
|
@@ -655,6 +655,7 @@ files:
|
|
|
655
655
|
- lib/labkit/rate_limit/config/error.rb
|
|
656
656
|
- lib/labkit/rate_limit/config/parser.rb
|
|
657
657
|
- lib/labkit/rate_limit/config/proto/config_pb.rb
|
|
658
|
+
- lib/labkit/rate_limit/config/schema.rb
|
|
658
659
|
- lib/labkit/rate_limit/config/vendored_proto.rb
|
|
659
660
|
- lib/labkit/rate_limit/configuration.rb
|
|
660
661
|
- lib/labkit/rate_limit/evaluator.rb
|